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Teat  and  Evaluation 
SOFTWARE  OPERATIONAL  ASSESSMENT  GUIDE 


This  pamphlet  describes  the  method  and  procedures  used  by  the  Air  Force  Operational  Test  and 
Evaluation  Center  (AFOTEC)  for  performing  software  operational  assessments  (SOA)  for  both 
embedded  computer  s3rstems  and  traditional  stand-alone  computer  systems. 

AFOTEC  Pamphlet  99-102  replaces  AFOTEC  Pamphlet  800-2,  all  volumes.  This  volume  is  one  in  a 
series  of  software  operational  test  and  evaluation  (OT&E)  guidelines  prepared  by  the  Software 
Analysis  Team  of  the  AFOTEC  Systems  Analysis  Directorate.  It  is  intended  for  use  in  operational 
test  and  evaluation  of  software.  Local  reproduction  of  all  volumes  in  this  series  is  authorized.  Direct 
any  comments  to  the  office  of  primary  responsibility  (OPR).  The  series  is  as  follows. 

AFOTEC  Pamphlet  99-102,  Volume  1  -  Management  of  Software  Operational  Test  and 

Evaluation 

AFOTEC  Pamphlet  99-102,  Volume  2  -  Software  Support  life  Cycle  Process  Evaluation 

Guide 

AFOTEC  Pamphlet  99-102,  Volume  3  -  Software  Maintainability  Evaluation  Guide 
AFOTEC  Pamphlet  99-102,  Volume  4  -  Software  Usability  Evaluation  Guide 
AFOTEC  Pamphlet  99-102,  Volume  5  -  Software  Support  Resources  Evaluation  Guide 
AFOTEC  Pamphlet  99-102,  Volume  6  -  Software  Maturity  Assessment  Guide 
AFOTEC  Pamphlet  99-102,  Volume  7  -  Software  Reliability  Evaluation  Guide 
AFOTEC  Pamphlet  99-102,  Volume  8  -  Software  Operational  Assessment  Guide 

1.  Purpose.  SOAs  are  an  integral  part  of  a  system's  operational  assessment  (OA).  The  central 
theme  of  an  OA  is  interaction  with  the  operating  command  and  developer  to: 

1.1.  Provide  a  perspective  on  a  system's  progress  toward  operational  effectiveness  and  suitability 
and  to  determine  if  there  are  deficiencies  or  voids  that  have  a  major  impact  on  operational 
effectiveness  gr»fl  suitability.  Using  development  test  data,  we  will  be  able  to  assess  how  the  system 
is  progressing  toward  meeting  user  requirements.  Or: 

1.2.  Assess  a  system's  readiness  for  OT&E. 

To  achieve  this  goal,  this  guide  describes  techmques  for  the  AFOTEC  software  test  manager  (STM) 
and  deputy  for  software  evaluation  (DSE)  to  use  in  assessing  aspects  of  computer  resources  and 
software  Hnring  a  system  OA.  This  guide  provides  the  necessary  information  to  plan,  execute,  and 
report  the  SOA  regardless  of  the  system's  development  phase. 

2.  Document  Organization  and  Use. 

2.1.  The  following  pages  describe  how  to  use  this  guide  in  planning,  executing,  and  reporting  the 
SOA  and  provide: 

OPR:  SAS  (Capt  Delmar  Gassner)  Certified  by:  SA  (Col  D.L.  BrazU) 

pi«»  **'’'‘‘*  Pages:  34/Distribution: 
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2.1.1.  A  background  of  the  AFOTEC  SOA 
methodology. 

2.1.2.  A  basic  understanding  of  the  assess¬ 
ment  procedures. 

2.1.3.  Attachment  1,  which  describes  the  as¬ 
sessment  factors  and  subfactors  for  use  in  any 
assessment  area. 

2.2.  This  guide  will  help  you: 

2.2.1.  Define  the  SOA  objectives. 

2.2.2.  Select  the  factors  relevant  to  the  SOA. 

2.2.3.  Collect  and  analyze  background  and 
supporting  data. 

2.2.4.  Assign  ratings  to  the  factors. 

2.2.5.  Analyze  and  interpret  the  results  to 
determine  the  rating  for  each  assessment 
area. 

2.2.6.  Report  the  results. 

Prior  to  beginning  the  SOA,  review  AFOTEC 
Pamphlet  (AFOTECPAM)  99-102,  volume  1  to 
gain  an  understanding  of  AFOTEC's  software 
OT&E  process. 

3.  Operational  Assessment  Areas.  There 
are  five  standsurd  areas  to  assess  during  the 
OA  of  a  target  program.  The  SOA  assessment 
areas  mirror  the  OA  assessment  areas: 

•  Area  1.  Major  impacts  affecting  opera¬ 
tional  effectiveness  and  suitability. 

•  Area  2.  Programmatic  voids. 

•  Area  3.  Adequacy  of  requirements. 

•  Area  4.  Ability  to  support  adequate 
operational  testing. 

•  Area  5.  Results  fi-om  higher  headquar¬ 
ters-directed  system  assessment  activities. 

Area  1,  where  we  examine  developmental  or 
other  relevant  test  data,  is  required  for  all 
SOAs,  with  the  possible  exception  of  higher 
headquarters-directed  specific  assessments 


(area  5).  The  assessment  areas  are  introduced 
in  the  following  subsections  (see  AFOTEC 
Instruction  (AFOTECI)  99-101  for  further 
clarification). 

3.1.  Area  1  (Migor  Impacts). 

3.1.1.  Scope.  Identify  and  assess  major  im¬ 
pacts  affecting  operational  effectiveness  and 
suitability. 

3.1.2.  Assess: 

3. 1.2.1.  Areas  of  risk  (known  areas  of  risk  in 
developing  the  system). 

3. 1.2.2.  Results  of  any  software  testing  to 
identify  significant  trends  noted  in  develop¬ 
ment  efforts. 

3.2.  Area  2  (Programmatic  Voids). 

3.2.1.  Scope.  Identify  any  software  or  com¬ 
puter  resomrce  programmatic  voids  that  would 
adversely  impact  the  ability  of  the  system  to 
meet  operational  requirements. 

3.2.2.  Assess.  The  program  information/ 
documentation  to  determine  if  software  devel¬ 
opment/testing  progresses  as  planned,  the 
system  will  meet  operational  requirements. 

3.3.  Area  3  (Adequacy  of  Requirements). 

3.3.1.  Scope.  Review  the  status  of  software 
documentation,  with  emphasis  on  user  re¬ 
quirements  development  including  complete¬ 
ness,  clarity,  sufficiency,  priority,  rationale,  or 
other  factors  that  could  affect  software  test¬ 
ability. 

3.3.2.  Assess.  The  software-related  docu¬ 
ments  to  determine  if  the  required  software 
and  computer  resource  information  is  con¬ 
tained  therein.  The  absence  or  inadequacy  of 
critical  documents  will  seriously  impact  the 
program. 

3.4.  Area  4  (Ability  to  Support  Adequate 
Operational  Testing). 

3.4.1.  Scope.  Review  the  schedule  for  the 
OT&E  events. 
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3.4.2.  Assess: 

3.4.2. 1.  Software  schedule  to  accommodate 
OT&E  activities. 

3.4.2.2.  Software  development/major  prob¬ 
lems  that  could  prohibit  conduct  of  OT&E  on  a 
properly  configured  test  article. 

3.4.2.3.  Required  software  and  computer 
resources  to  support  OT&E. 

3.4.2.4.  Software  development  and  maturity 
aspects  that  would  impact  OT&E. 

3.5.  Area  5  (Higher  Headquarters  Re¬ 
sults). 

3.5.1.  Scope.  Review/observe  special  field 
test  activities  or  other  efforts  when  specifically 
requested. 

3.5.2.  Assess.  Higher  headquarters-directed 
software  and  computer  resource  issues. 

4.  Planning.  The  following  considerations 
are  necessary  to  plan  an  effective  SOA: 

4.1.  Select  Relevant  Factors.  You  must 
clearly  define  the  procedures  for  performing 
the  SOA.  To  do  this,  identify  the  factors,  tools, 
and  program  data  resources  relevant  to  a 
specific  program  assessment.  The  SOA  team 
^ould  begin  by  reviewing  attachment  1  to 
select  the  factors  relevant  to  the  assessment  of 
each  of  the  areas.  Eliminate  factors  or  subfac¬ 
tors  that  are  unnecessary  or  premature  for  the 
current  program  stage  of  development.  Also 
review  the  statement  of  work  (SOW)  and 
contract  data  requirements  lists  (CDRL)  to 
understand  the  required  design,  development, 
testing,  and  quality  processes. 

4.2.  SOA  Resoinrces  and  Other  Considera¬ 
tions.  There  will  be  constraints  on  the  SOA 
team  in  terms  of  time,  funds,  and  personnel. 
Consider  the  following  when  developing  the 
SOA  plan: 

4.2.1.  Personnel  Resources.  In  most  cases, 
the  STM  or  DSE,  using  AFOTEC  resources, 
will  accomplish  the  SOA  The  combination  of 
personnel  experience,  time  allowed,  and  pro 


gram  size  will  determine  the  breadth  and 
depth  of  the  SOA. 

4.2.2.  Program  Data  Resources. 

4.2.2.1.  Review  program  status  information  to 
include  government  and  deliverable  documen¬ 
tation,  program  reviews,  and  interviews. 

4.2.2.2.  Team  members  should  attend  any 
program  reviews  prior  to,  or  falling  within  the 
time  frame  of,  the  SOA,  and  use  key  program 
personnel  as  information  sources. 

4.2.2.3.  If  contractor  internal  data  resources 
(such  as  software  development  files  (SDF))  are 
included  in  the  program  resomces,  arrange 
with  the  program  office  in  advance  to  gain 
access. 

4.2.2.4.  Previous  evaluations/assessments  may 
provide  insight  into  problem  areas  or  program 
voids.  The  SOA  team  must  review  any  other 
studies  performed,  such  as  previous  SOAs, 
development  test  and  evaluation  (DT&E), 
IV&V  or  Software  Engineering  Institute  (SEI) 
assessments  to  gain  information  useful  to  the 
SOA  effort. 

4.2.3.  Other  Considerations: 

4.2.3. 1.  Identify  travel  and  funding  require¬ 
ments  (for  data  collection,  conducting  inter¬ 
views,  attending  reviews  and  computer  sup¬ 
port)  in  the  test  resource  plan  (TRP). 

4.2.3.2.  Contact  any  necessary  supporting 
agencies. 

4.2.3.3.  Allow  for  special  securily  considera¬ 
tions. 

6.  SOA  Execution.  Responses  to  the  factors 
and  subfactors  in  attachment  1  are  the  core  of 
the  SOA  Use  other  tools  and  information 
sources  to  provide  backgroxmd  information 
and  determine  if  problem  areas  are  receiving 
adequate  attention. 

5.1.  SOA  Rating  Scale. 

5.1.1.  Green.  No  issues  or  issues  receiving 
adequate  attention. 
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5.1.2.  Yellow.  Issues  requiring  additional 
attention. 

5.1.3.  Red.  Significant  areas  of  concern  or 
voids  exist  in  schedule,  resources,  system 
flosign,  etc.,  that  will  impact  OT&E  and/or 
may  prevent  the  achievement  of  specified 
operational  requirements. 

5.1.4.  White.  Area  not  assessed. 

5.2.  Recording  Responses.  Your  responses 
will  depend  on  your  eaperience  and  subjective 
judgment.  If  you  score  a  factor  or  subfactor 
yellow  or  red,  record  the  impact  of  such  a 
rating  (major,  minor,  none)  and  the  status  of 
any  plan  to  correct  the  deficiency  (plan  in- 
place,  plan  in-place  but  inadequate,  no  plan). 
Finally,  you  must  record  the  rationale  for  a 
low  rating  for  a  factor  or  subfactor.  Ako 
record  supplementary  information  to  provide 
historical  information. 

5.3.  Ratings.  Assign  a  color  rating  to  each 
subfactor.  Combine  the  subfactor  ratings  into 
a  single  color  rating  for  the  parent  factor  and 
then  combine  the  factor  ratings  into  a  single 
color  for  the  parent  assessment  area.  Again, 
use  your  best  judgment  coupled  with 
comments  and  other  information  recorded  for 
each  factor  to  make  a  decision  as  to  the  overall 
color  rating  for  an  assessment  aiaa. 

6.  SOA  Inputs  to  the  OA  Plan/Report. 

6.1.  The  customer  for  all  SOA  inputs  and 
results  is  the  program's  operational  test  man¬ 
ager.  The  test  manager  will  determine  the 
format  for  each  plan  and  report.  The  SOA 
team  will  be  responsible  for  providing  wording 
on  the  following: 

6.1.1.  Software  and  computer  resource  de¬ 
scription. 

6.1.2.  How  the  software  SOA  fits  into  the 
system  OA 

6.1.3.  SOA  dates  and  location. 

6.1.4.  The  data  sources  that  were  selected  to 
address  tiie  software  and  the  computer 


resource  measures  of  performance  (MOP)  and 
measures  of  effectiveness  (MOE). 

6.1.5.  The  user  requirements  for  the  software 
and  computer  resource  MOPs  and  MOEs. 

6.1.6.  How  test  data  results  can  be  utilized  to 
examine  software  and  computer  resource 
MOPs  and  MOEs  under  a  specific  COI  and 
draw  effectiveness  and  suitabilily  conclusions. 

6.1.7.  Anticipated  planning  considerations 
and  limitations  that  may  affect  the  SOA  con¬ 
cept  or  scope. 

6.1.8.  list  those  areas  and  factors  to  be  exam¬ 
ined  during  the  SOA.  Keep  in  mind  that  we 
include  Area  1  in  every  SOA. 

6.1.9.  Provide  an  overview  of  how  the  SOA 
will  be  or  was  conducted. 

6.1.10.  List  planning  considerations  and 
limitations  that  shaped  the  SOA  design  or 
affected  assessment  results. 

6.1.11.  State  overall  conclusions  with  a  list  of 
recommendations  (if  appropriate). 

6.2.  After  conclusion  of  Ihe  SOA,  include  any 
lessons-learned  in  SAS's  SOA  database. 

7.  Summary  of  Assessment  Philosophy. 
The  following  is  a  summary  of  the  general 
philosophy  to  guide  you  in  assessing  the  system: 

7.1.  Your  primary  consideration:  "Are  the 
system's  software  and  computer  resource  pro¬ 
gressing  toward  operational  effectiveness  and 
suitability,  and  are  these  functions  ready  to 
enter  the  next  phase  of  the  life  cycle?" 

7.2.  Some  necessary  resources  may  not  be 
available  at  the  time  of  the  SOA.  You  must 
tailor  out  those  areas  which  do  not  apply  to 
the  system  or  are  not  available.  However,  if 
items  are  not  available  that  should  be 
available  at  that  point  in  the  system's  life 
<ycle,  you  should  rate  them  lower.  . 

7.3.  Finally,  bear  in  mind  you  were  chosen  for 
this  SOA  because  of  your  demonstrated 
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expertise.  That  expertise  and  the  profes-  assessment  will  help  provide  the  Air  Force 

sionalism  you  demonstrate  in  completing  this  with  a  quality  software  product. 


GEORGE  B.  HARRISON,  Major  General,  USAF 
Commander 

1  Attachment 

1.  Software  Operational  Assessment  (SOA)  Factors. 
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SOFTWARE  OPERATIONAL  ASSESSMENT  (SOA)  FACTORS 


INTRODUCTION 

This  attachment  containa  a  list  of  factors  and  subfactors  you  can 

moot  areas  AFOTECPAM  99-102,  volume  2,  attachment  3  contams  a  hst  of  the  Software  Suppo 
Sfe  vestions  that  should  be  answerable  at  the  various  time  i^ses  or  mJestoM 

“eS  ThTS.^  are  grouped  in  five  categories  (NOTE:  The  categories  are  just  placehelders  to 

group  related  factors): 


1.  Program  Schedule  Adequacy. 

2.  Program  Resources. 

3.  Documentation  Status, 


4.  Development/Maturity. 

5.  Requirements  Traceability. 

Assess  only  those  factors  and  subfactors  applicable  to  this  SOA.  You  can  deteWe  rating  at  ^e 
factor  level  or  the  subfactor  level.  The  factor  and  subfactors  are  statements  desOTbmg  desirable 
iSLtes  of  a  p^gram's  development  and  testing  p^cesses.  When  respo^e  ^ 

provided,  respond  to  the  factors  and  subfactors  based  on  your  expenence  and  expertise.  Yo  y 
tailor  the  individual  factors  and  subfactors  or  add  factors/subfactors  of  your  choice. 

listed  below  are  the  categories,  factors,  and  subfactors  available. 


1.  Program  Schedule  Adequacy: 

1.1.  Development  Schedule  Adequacy. 

1.2.  OT&E  Schedule  Adequacy. 


2.  Program  Resources: 

2.1.  Test  Article  Availability- 

2.2.  Test  Resources. 


2.3.  CRWG. 

3.  Documentation  Status: 

3.1.  planning  Documents: 

3.1.1.  TEMP. 

3.1.2.  OT&E  Plan. 


3.1.3.  TRP. 


3.1.4.  CRLCMP. 
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3.2.  Requirements  Documents: 

3.2.1.  ORD. 

3.2.2.  RCM. 

4.  Development/Matiuily: 

4.1.  Software  Development  Processes: 

4.1.1.  Audits/Reviews. 

4.1.2.  Configuration  Management: 

4.1.2.1.  Identification. 

4.1.2.2.  Control. 

4.1.2.3.  Audit. 

4.1.2.4.  Accounting. 

4.1.3.  Quality  Assm-ance. 

4.1.4.  Program  Management: 

4.1.4.1.  Resource  Fimctions. 

/ 

4.1.4.1.1.  Planning. 

4. 1.4. 1.2.  Organization. 

4.1.4.1.3.  Monitoring. 

4.I.4.2.  Control  Functions. 

4.1.4.2.1.  Corporate  Policies. 

4.1.4.2.2.  Subcontractor  Management. 

4.1.4.2.3.  Communication  Interfaces. 

4.1.5.  Testing. 

4.2.  Products: 

4.2.1.  Documents. 

4.2.1.1.  SDP. 

4.2.1.2.  SSDD. 

4.2.1.3.  IDD. 


4.2.I.4.  SQPP. 
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4.2.1.5.  SCMP. 

4.2.1.6.  SDD. 

4.2.1.7.  STP. 

4.2.1.8.  SRS. 

4.2.1.9.  IRS. 

4.2.2.  Software. 

4.2.2.1.  Reliability. 

4.2.2.2.  Schedules. 

4.2.2.3.  Spare  Capacity. 

4.2.2.4.  Maintainability. 


5.  Requirements  Traceability: 

5.1.  ORD/RCM  to  System  Specification  Traceability. 


5.2.  Development  Contractor  Traceability- 


1.  PROGRAM  schedule:  ADEQUACY 


The  program's  master  schedule  must  clearly  show  schedules  adequate  to  support  computer  re^ce 
DT&E  and  OT&E.  The  master  schedule  must  incorporate  current  status  mformation  of  tee  so^ar 
development  schedule.  You  need  to  know  what  is  happening  on  a  broader  scale  with  sof^^e  devel- 
opm^teesting  and  how  identified  risk  areas  might  impact  tee  abihty  to  meet  program  schedules. 


FACTOR:  The  program’s  master  schedule  is  adequate  to  support  computer  resource  development, 
DT&E,  and  OT&E. 


SUBFACTORS: 

DEVELOPMENT  SCHEDULE  ADEQUACY  (1.1)  -  The  software  development  schedule  is 
adequate. 

_ ^OT&E  SCHEDULE  ADEQUACY  (1.2)  -  The  OT&E  schedule  is  adequate  to  support  computer 

resoxirces  and  software  evaluations. 

FYPT.ANATTON:  Many  activities  must  be  identified  and  shown  on  tee  schedule  to  ensure  they  are 
tracked  for  proper  completion. 

RFFF.RFNCE  QTTFSTTONS  (NOTE  -  see  AFOTECPAM  99-102.  volume  2.  Softwate  Life  Cyd£ 
Prncflsfi  Evaluation  Guide  for  emansinn  of  the  reference  OWStlOBS): 

SPM(PL)-3  Planning  for  computer  resources  has  been  based  on  an  acquisition  schedule  with 

adequately  specified  nulestenes.  .  .i.  v  + 

SPM(PL)-16  Planning  for  DT&E  of  computer  resources  has  been  adequate  throughout  toe 
system  life  cycle. 
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_ SPM(0S)-1  The  software  requirements  have  been  adequately  allocated  to  elements  of  work 

breakdown  structure  (WBS). 

_ SPM(OS)-2  The  software  related  tasks  are  clearly  identified  in  the  WBS. 

SPM(OS)-3  The  key  project  personnel  and  their  assignments  in  relation  to  the  WBS  software- 
related  tasks  are  clearly  identified. 

_ SPM(OS)-4  The  coordination  of  modifications  to  the  WBS  among  all  activities  has  been 

adequate. 

_ SPM(TS)-6  The  requirements  for  the  development  contractor  software  test  strategy  are  clearly 

specified  in  the  RFP,  SOW,  and/or  CDRLs. 

1.1.  DEVELOPMENT  SCHEDULE  ADEQUACY 
FACTOR:  The  software  development  schedule  is  adequate. 

FYPT.ANATTON:  It  is  critical  the  software  development  work  meets  the  overall  schedule.  Specifi- 
investigate  the  progress  on  software-related  tasks,  but  also  identify  any  preceding  tasks  upon 
which  the  software  development  activity  depends.  The  Software  Development  Plan  will  document 
the  software  development  schedule. 

-  The  WBS  should  include  computer  resource  activities  such  as  major  reviews  and  milestones 
and  be  used  as  the  basis  for  schedule  and  cost  estimates.  The  WBS  should  be  updated 
throughout  the  program  to  reflect  significant  changes. 

-  The  critical  path  of  the  schedule  should  be  identified. 

-  Confirm  each  computer  resource  schedule  milestone  is  a  measurable  event. 

-  The  SPO  should  have  developed  and  exercised  a  software  risk  abatement  plan. 

.  The  program  schedule  must  also  include  adequate  time  for  DT&E.  In  addition  to  time  to 
execute  DT&E,  the  program  schedule  should  include  sufficient  time  for  any  system  rework 
that  is  required  based  on  DT&E  findings. 

BF.c!PnTsrSE  INSTRUCTIONS:  The  evaluator  should  look  for  impacts  which  have  not  been  reflected 
in  the  current  schedule  and,  therefore,  make  it  unrealistic  or  unachievable.  For  the  current 
schedule,  if  all  significant  major  milestones  or  key  software  deliverable  dates  are  slipping,  or  have 
slipped,  or  show  signs  of  converging  to  unrealistic  dates,  the  evaluator  should  consider  assigning  a 
red  to  this  factor. 

RFT.ATFn  METRICS:  Examine  a  schedule  metric  that  indicates  adherence  to  the  planned  schedules 
for  mqjor  software  milestones  and  key  software  dehverables.  Compute  the  planned  and  actual 
schedules  for  major  milestones  and  key  software  deliverables  as  they  change  over  time.  (See  APT 
800-48.) 

1.2.  OT&E  SCHEDULE  ADEQUACY 

FACTOR:  The  OT&E  schedule  is  adequate  to  support  the  computer  resources  and  software  evalu¬ 
ations. 

FYPT.ANATION:  Tnitinl  information  on  activities  to  be  performed  during  the  OT&E  may  be  obtained 
from  the  test  concept  and  the  TEMP.  The  test  concept  should  identify  an  initial  OT&E  schedule. 
The  OT&E  plan  should  contain  test  objectives  and  evaluation  methods  to  assess  the  software.  The 
schedule  should  include  activities  for  both  system  effectiveness  (i.e.,  system  performance  objectives) 
and  system  suitability  evaluations  (i.e.,  maintainability).  Obtain  the  adequacy  of  the  schedule  to 
pggAgg  system  effectiveness  and  suitability  by  examining  test  mission  scenarios  and  the  Data  Man- 
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aeement  and  Analysis  Plan  (DMAP)  for  tests  that  include  software  components.  Assess  the  software 
aspects  of  these  tests  and  cross-reference  them  to  the  functional  software  requirements  of  the  sys¬ 
tem. 


In  addition  to  time  allocated  for  OT&E,  there  should  be  sufficient  slack  time  so  any  slippage  of 
preceding  activities  (such  as  DT&E)  does  not  impact  OT&E. 


reference  QUESTION: 

_ SPM(PL)-17  Planning  for  OT&E  of  computer  resources  has  been  adequate  throughout  the  sys 

tern  life  cycle. 


2.  PROGRAM  RESOURCES 


Resources  required  by  the  program  must  be  addressed  in  the  program  s  plannmg  process 
should  account  for  the  availability  of  aU  components  such  as  personnel,  equi^ent,  ^d  facihties 
required  to  meet  the  schedule  and  describe  the  special  characteristics  of  software  development. 
Planning  should  identify  specific  test  articles  and  test  resources  and  ensure  their  Wabihfy.  to 
addition,  the  presence  of  the  CRWG,  which  coordinates  many  of  the  processes  associated  with  soft¬ 
ware  issues,  must  be  addressed. 


FACTOR:  The  plaimed  computer  resources  are  adequate  to  support  software  development,  DT&E, 
and  OT&E. 


SUBFACTORS: 

__TEST  ARTICLE  AVAILABILITY  (2.1)  -  The  required  test  articles  are  planned  to  be  available  for 
PT&F-  and  OT&E  of  computer  resources. 

test  resources  (2.2)  -  The  test  computer  resources  required  to  support  the  software  testing 
objectives  are  adequately  planned  and  will  be  available  for  DT&E  and  OT&E. 


.CRWG  (2.3)  -  The  CRWG  is  operational. 


EYPT . ANATION:  The  successful  completion  of  DT&E  and  OT&E  requires  the  avafiabOily  of  identi- 
fi^^  conSed  software  test  articles.  In  addition,  planning  must  occur  for  toe  availabihty  of 
hardware  and  software  as  well  as  evaluation  tools  and  personnel  to  perform  toe  evaluations. 


2.1.  TEST  ARTICLE  AVAILABILITY 

FACTOR:  The  required  test  articles  will  be  available  for  DT&E  and  OT&E  of  computer  resources. 

EYPT, ANATION*  The  primary  sources  for  information  on  what  software  producte  should  be  provided 
Statement  of  Work  (SOW),  toe  program  CDRL.  and  toe  OT&E  plan^  A  hst  of  test 
article  items,  or  "first  article  test  items,"  should  be  provided  in  these  documents.  Once  they  are 
identified,  you  should  compare  toe  software  test  articles  with  a  schedule  for  toeir  dehveiy  and 
ensure  the  schedules  for  software  development,  DT&E,  and  OT&E  indicate  toe  software  test  articles 
will  be  available  by  toe  need  dates. 

For  OT&E  to  be  successfiilly  completed,  toe  software  products  must  be  complete,  cer^ed,  and 
provided  in  toe  quantities  required.  Software  should  be  patch-fi:ee  and 

Lency  prior  to  use  during  toe  OT&E.  All  software  products  provided  for  OT&E  should  be  imder 
sMct  software  configuration  control  by  the  contractor.  Plans  for  these  activities  should  be  docu- 
mented  in  the  SOW  and  the  SDP . 
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Included  in  the  definition  of  software  test  articles  are  all  deliverables  (see  the  CDRLs)  requir^for 
the  system,  to  include  software  documents  required  to  operate  and  maintain  the  system.  These 
articles  will  be  subjected  to  supportability  evaluations  during  the  OT&E.  Reference  the 
development/maturity  category  for  details  on  determining  the  system  development/matunty  issues. 

2^.  TEST  RESOURCES 

FACTOR:  The  test  computer  resources  required  to  support  the  software  testing  objectives  are  ade¬ 
quately  planned  and  will  be  available  for  DT&E  and  OT&E. 

FYPT . AN ATION:  Plans  must  exist  describing  the  hardware  on  which  the  software  to  be  tested 
resides  In  addition  to  the  software  under  development,  any  additional  support  software  and  data¬ 
bases  required  for  testing  need  to  be  available,  populated  with  the  required  data,  and  vahdated. 
Capabilities  and  limitations  of  aU  software  tools  (and  hardware  resources,  mterfaces,  etc.)  need  to  be 
documented  and  available  to  the  test  team. 

Use  the  test  concept,  TRP,  the  CDRL,  the  OT&E  plan,  and  the  SDP  to  identify  test  resources.  The 
OT&E  plan  should  include  specific  evaluation  methods  for  software.  This  plan  should  specify  what 
type  of  evaluations  are  planned  and  what  resources  are  required  for  each  one.  Other  sources  of 
information  may  be  agreements  between  AFOTEC,  the  SPO,  and  using  commands. 

REFERENCE  QUESTION: 

SPM(TS)-18  The  software  -  test  environment  (host  emulations/simulation,  inte^ated 
laboratory  environments,  software  bench  testing  equipment,  and  perational 
mission  test  environments)  is  adequately  identified  in  the  software  test 
documentation  and  is  adequate  for  accomplishing  the  required  testing. 

2.3.  CRWG 

FACTOR:  The  CRWG  is  operational. 

EVPT . ANATION:  The  CRWG  should  be  established  prior  to  Milestone  I.  Its  function  is  to  proride 
guidance  on  computer  resources  issues  throughout  the  development  effort.  The  CRWG  is  responsible 
for  innlring  key  recommendations  to  the  program  manager  and  for  describing  the  support  concept  m 
the  CRLCMP.  It  should  include  members  firom  the  operating,  supporting,  and  participating  com¬ 
mands. 

The  CRWG  participates  in  program  management  reviews,  source  selection  evaluation  boards,  solici¬ 
tation  review,  design  reviews,  and  audits.  Actions  of  the  CRWG  may  help  in  identifying  a^^ties 
that  should  occur  for  the  system  to  be  ready  for  OT&E.  The  CRWG  also  supports  the  ^WG  ^ 
required.  The  CRWG  should  exist,  and  it  should  be  active.  CRWG  meeting  minutes  should  be 

available  for  examination. 

RESPONSE  INSTRUOTTONS:  If  a  CRWG  has  not  been  established,  or  is  not  meeting  on  a  periodic 
basis,  the  evaluator  should  consider  rating  this  area  as  red. 

REFERENCE  QUESTIONS: 

SPM(PL)-12  The  CRWG  organization  has  been  adequate  throughout  the  system  life  cycle. 

_ ^SPM(PL)-13  The  CRWG  has  had  clearly  specified  responsibilities  and  appropriate  authority  to 

implement  those  responsibilities  throughout  the  system  life  cycle. 

SPM(PL)-14  The  CRWG  has  properly  ensured  that  computer  resources  comply  with 
established  policy,  procedures,  plans  and  standards. 

SPM(PI)-9  The  CRWG  external  interfaces  are  adequate. 
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3.  DOCUMENTATION  STATUS 

The  objective  of  the  documentation  status  factors  is  to  review  the  critic^  documents  (TEMP,  OT&E 
Dlan  TRP  and  CRLCMP)  to  determine  whether  they  have  progressed  sufficiently  to  support  ffie 
DT&E  and  OT&E.  In  addition,  the  ORD  and  RCM  are  reviewed  to  determme  ffie  adequacy  of  the 
requirements,  specifications,  and  test  criteria  (e.g.,  prioritized,  operationally  stated,  no  disconnects). 

FACTOR:  The  documentation  is  progressing  adequately  to  support  DT&E  and  OT&E  of  computer 
resources. 


SUBFACTORS: 

PLANNING  DOCUMENTS  (3.1)  -  The  planning  documents  are  progressing  adequately  to  sup- 
port  OT&E  of  computer  resources. 

_ requirements  DOCUMENTS  (3.2)  -  The  requirements  documents  are  progressing  ade¬ 
quately  to  support  OT&E  of  computer  resources. 

3.1.  PLANNING  DOCUMENTS 

FACTOR:  The  planning  documents  are  progressing  adequately  to  support  DT&E  and  OT&E  of 
computer  resources. 


STTBFACTORS: 

TEMP  (3.1.1)  -  The  TEMP  contains  the  appropriate  computer  resource  information. 

OT&E  PLAN  (3.1.2)  -  The  OT&E  plan  contains  the  appropriate  computer  resource  information, 


TRP  (3.1.3)  -  The  TRP  includes  the  appropriate  computer  resource  information. 

_ CRLCMP  (3.1.4)  -  The  CRLCMP  contains  the  appropriate  computer  resource  information. 

FyPT.ANATTON:  Factors  have  been  included  for  the  foUowing  planning  dociments:  TOW,  OT&E 

pl^,T^  and  CRLCMP.  Additional  (planning)  documents  ^t  can  be  reviewed 

Lility  and  maintainability  management  plan,  MOAMOU,  techmcal  orders,  and  the  Jomt  Rehabihty 

and  Maintainabilily  Evaluation  Team  ( JRMET)  charter. 

The  documents  should  be  updated  when  there  are  significant  changes  to  ffie  program,  at  milestone^ 
and  at  other  m^or  events.  There  should  be  a  process  in  place  to  ensure  aU  documents  are  updated  to 

reflect  changes  in  the  requirements. 


3.1.1.  TEMP 

FACTOR:  The  TEMP  contains  the  appropriate  computer  resources  information. 

fypt.ANATION*  The  TEMP  should  summarize  the  critical  computer  resource  operational  effective- 
ness  md  suitability  parameters.  There  should  be  no  disconnects  in  computer  resource  requirements 
between  the  ORD,  RCM,  and  the  TEMP. 

The  TEMP  should  list  the  critical  software  technical  parameters  of  the  system  w^  the 
panying  objectives  and  thresholds.  These  critical  technical  parameters  ^dmved  froin  the  ORD, 
^tSl  system  characteristics,  and  technical  performance  measures.  The  TOW  shoifid  also  adless 
support  concepts  (i.e.,  postdeployment  software  support)  resulting  m  special  test  and  analysis 

requirements. 
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The  TEMP  should  address  computer  resources  testing  to  include  the  level  of  stress  testing  that  will 
be  required  (e.g.,  out  of  range,  load,  alternative  sign,  maximum  and  minimum  value). 

REFERENCE  QUESTIONS: 

_ ^SPM(PL)-1  Planning  for  computer  resources  has  been  adequate  with  respect  to  acquisition 

development,  logistics,  and  training. 

_ ^SPM(TS)-3  The  software  test  process  for  OT&E  has  followed  the  guidelines  in  the  TEMP. 

3.1^.  OT&E  PLAN 

FACTOR:  The  OT&E  plaTi  contains  the  appropriate  computer  resource  information. 

EyPTANATION:  The  OT&E  plan  should  contain  the  appropriate  computer  resources  evaluation 
criteria.  These  raiteria  are  tied  directly  to  the  operating  command’s  stated  computer  resource  opera¬ 
tional  requirements  and  represent  a  level  of  performance  against  which  system  characteristics  are 

compared. 

The  computer  resource  objectives  that  are  addressed  in  the  OT&E  plan  should  be  consistent  with  &e 
computer  resources  information  in  other  documents  that  address  this  type  of  information 
(specifically  the  TRP  and  TEMP). 

The  OT&E  plsu  should  spell  out  the  software  test  readiness  criteria.  AFOTEC/SAS  has  develop^  a 
boilerplate  of  typical  software  OT&E  test  readiness  requirements  for  a  program  that  is  certified 
ready  to  begin  OT&E.  If  the  OT&E  plan  is  not  available,  review  the  test  concept  or  Test  Process 

Review  (TPR)  notes. 

RESPONSE  TN.t}TR.TTnTTONS:  Cross-check  the  OT&E  evaluation  criteria  with  those  in  the  RCM 
and  with  the  ORD  requirements  to  ensure  completeness,  applicability,  and  reasonableness,  .^y 
disagreements  between  the  documents  should  be  noted  and  the  factor  rated  yellow  or  red  depending 
upon  the  severity  of  the  disconnect. 

3.1.3.  TRP 

FACTOR:  The  TRP  includes  the  appropriate  computer  resource  information. 

EYPT.ANATION:  The  TRP  should  identity  requirements  for  computer  resources  to  include  hard¬ 
ware,  software?,  supplies,  and  personnel  (e.g.,  evaluators,  DSE). 

All  software  evaluation  personnel  must  be  identified  in  the  TRP.  This  identification  must  be  speci¬ 
fied  for  each  fiscal  year  and  include  the  title,  grade,  number,  source  (i.e.,  AFM(3),  number  of  days  per 
person,  dates  required,  site,  and  t^e  of  assignment  (e.g.,  TDY,  PCS).  The  training  for  the  DSE  an 
software  evaluator  should  also  be  identified  in  the  TRP . 

3.1.4.  CRLCMP 

FACTOR:  The  CRLCMP  contains  the  appropriate  computer  resoxirce  information  required  to  sup¬ 
port  DT&E  and  OT&E. 

EXPT.ANATION:  The  CRLCMP  should  describe  the  functional  interrelationships  among  the  follow¬ 
ing  organizations:  implementing  command,  using  command,  participating  commands  (testing, 
training,  and  contract  administration),  and  other  agencies.  These  descriptions  should  include  an 
identifi«»tinTi  of  the  major  functions  and  responsibilities  that  relate  to  computer  resources.  CRLCMP 
requirements  are  listed  in  AFR  800-14. 
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pF-SPONST;  TMgTRTTnTTONS:  If  any  of  the  AFR  800-14  information  is  missing  from  the  CRI^MP, 
the  risk  of  problems  during  testing  is  higher.  Use  judgment  in  determining  if  missmg  information  is 
sufficient  reason  to  rate  this  factor  yellow  or  red. 


REFERENCE  QUESTIONS: 

SPM(PL)-9  The  CRLCMP  contains  adequate  specifications  of  the  acquisition  requirements  for 
computer  resources. 

SPM(PL)-10  The  CRLCMP  adequately  addresses  the  responsibilities  and  procedures  to  ensure 
proper  software  configuration  management  throughout  the  system  life  cycle. 

3.2.  REQUIREMENTS  DOCUMENTS 

FACTOR:  The  requirements  documents  are  progressing  adequately  to  support  DT&E  and  OT&E  of 
computer  resources. 


SUBFACTORS: 

ORD  (3.2.1)  -  The  ORD  has  reached  a  level  of  completion  consistent  with  tiie  schedule. 


__RCM  (3.2.2)  -  The  RCM  contains  the  appropriate  computer  resource  information. 

EXPLANATION:  The  requirements  documents  are  the  ORD  and  the  RCM.  These  dooments  should 
co^in  the  c^puter  resources  requirements.  These  requirements  should  be 
terms  and  be  testable  in  order  to  support  DT&E  and  OT&E.  (See  AFI 1-601  for  ORD/RCM  require¬ 
ments.) 


3.2.1.  ORD 


FACTOR:  The  ORD  has  reached  a  level  of  completion  consistent  with  the  schedule. 

EXPLANATION:  The  ORD  contains  the  operating  command's  documentation  of  essential  quantita¬ 
tive  and  qualitative  requirements  and  describes  how  the  system  will  be  used,  hs  requireme^ 
change  throughout  the  system  development  process,  the  ORD  should  be  updated  to  reflect  the 

changes. 

The  critical  computer  resource  requirements  described  in  the  ORD  appear  m  pa^eters  in  the 
RCM.  The  parameters  in  the  RCM  should  be  predicated  upon  requirements  m  tiie  ORD.  You  sh^d 
identify  the  critical  computer  resource  requirements  in  the  ORD  and  ensure  they  are  m  the  RCM. 
Use  the  following  template  for  requirements  that  should  be  in  the  ORD: 


(1)  Requirements  identifying  computer  resource  constraints  (e.g.,  language,  computer). 

(2)  Requirements  addressing  all  ILS  elements  for  computer  resources. 

(3)  Requirements  addressing  plans  for  using  command  support  of  software. 

(4)  Requirements  for  software  that  support  user-fidendly  operations  and  maintenance. 

(5)  Software  maturity  requirements.  ,  ^  ^ 

(6)  A  requirement  for  when  the  software  support  agency(s)  must  be  functional.  .... 

(7)  Requirements  for  high  quality  software  source  code,  documentation,  and  architecture 

design. 

(8)  Environmental  computer  resource  constraint 

(9)  Requirements  for  timings 

(10)  Requirements  for  spare  memory  and  throughput. 

(11)  Requirements  for  computer  memory  and  throughput  growffi. 

(12)  A  requirement  for  development  of  a  software  support  facihty. 

(13)  A  requirement  for  software  configuration  control. 
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(14)  Accuracy  requirements  for  each  applicable  function. 

(15)  Error  handling  requirements. 

(16)  Interface  capability  requirements. 

3.2.2.  RCM 

FACTOR:  The  RCM  contains  the  appropriate  computer  resource  information. 

F.XPTANAT10N:  The  RCM  documents  and  tracks  creation  of  and  changes  to  user  requirements.  Its 
purpose  is  to  provide  a  comparison  and  correlation  of  requirements  to  specifications  and  test  criteria. 
It  is  a  mandatory  attachment  to  the  ORD.  The  RCM  contains  a  comparison  of  the  user's  system 
requirements,  contractual  specifications,  and  operational  evaluation  criteria. 

An  overabundance  of  changes  to  the  RCM  may  indicate  instability  in  the  requirements.  The  RCM 
shows  current  requirements  or  goals  from  the  most  recent  ORD  side-by-side  with  those  from  any 
previous  ORD  to  provide  an  audit  trail  of  requirements  evaluation. 

The  parameters  in  the  RCM  must  be  stated  in  quantifiable  terms  so  requirements,  specifications, 
and  test  criteria  can  be  applied  to  them. 

Computer  resource  requirements  must  not  be  contradictory  (i.e.,  a  requirement  for  all  software  to  be 
written  in  a  high  order  language  and  another  requirement  for  flight  control  software  to  be  written  in 
assembly  language). 

The  computer  resomce  requirements  should  be  operationally  oriented  as  opposed  to  specification 
oriented. 

The  computer  resource  requirements  should  be  testable  so  test  cases  coiild  reasonably  be  set  up. 
REFERENCE  QUESTIONS: 

_ SPM(PL)-25  The  procurement  and  operational  support  planning  documents  have  been 

adequately  updated  as  living  documents  throughout  the  system  life  cycle. 

_ SPM(DM)-9  The  profile  of  changes  to  software  requirements  is  reasonable. 

4.  DEVELOPMENT/MATURITY 

Maturity  is  defined  in  AFOTECPAM  99-102,  volume  6,  as  a  "measure  [of]  the  software's  progress  in 
its  evolution  toward  satisfying  dociunented  user  requirements."  The  objective  of  these  factors  is  to 
determine  if  any  problems  exist  in  the  software  development  processes  or  products  that  will  impact 
DT&E  and  OT&E.  To  meet  this  objective,  you  must  assess  the  software  processes  and  the  software 
product  in  terms  of  development(maturity. 

Problems  in  the  software  development  processes  are  identified  by  assessing  the  audit  and  review, 
configuration  management,  quality  assurance,  program  management,  and  testing  processes.  Audits 
and  reviews  provide  insights  into  the  maturity  of  the  software  products.  Problems  in  the  software 
development  products  are  identified  by  assessing  the  software  documentation  and  source  code.  A 
review  of  the  software  documents  will  determine  their  level  of  completion  and  whether  they  contain 
the  appropriate  computer  resources  information.  A  review  of  the  source  code  will  reveal  whether  it 
is  progressing  sufficiently. 


FACTOR:  There  are  no  significant  development/maturity  problems  in  the  software  development 
effort  that  would  impact  DT&E  and  OT&E. 
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SUBFACTORS: 

_ SOFTWARE  DEVELOPMENT  PROCESSES  (4.1)  -  The  maturity  of  the  software  development 

processes  is  adequate  to  ensure  properly  configured  operational  software. 

_ ^SOFTWARE  DEVELOPMENT  PRODUCTS  (4.2)  -  Development  and  maturity  of  the  software 

products  are  adequate  to  ensure  properly  configured  operational  software. 

4.1.  SOFTWARE  DEVELOPMENT  PROCESSES 

FACTOR:  The  maturity  of  the  software  development  processes  is  adequate  to  ensure  properly  con¬ 
figured  operational  software. 

SUBFACTORS: 

AUDITS/REVIEWS  (4.1.1)  -  The  software  development  audit  and  review  processes  indicate 
development  of  a  maturing  product. 

_ CONFIGURATION  MANAGEMENT  (4.1.2)  -  The  software  configuration  management  (SCM) 

processes  are  adequate. 

QTTAT.TTV  ASSURANCE  (4.1.3)  -  The  software  quality  assimance  policies  and  procedures  are 
effective. 

PROGRAM  MANAGEMENT  (4.1.4)  -  Program  management  processes  in  place  to  determine 
implementation  methodologies,  assess  progress,  and  identify  shortfalls  are  adequate. 

TFRTTKG  (4.1.5)  -  The  testing  processes  are  adequate  to  ensure  software  is  being  tested  as 
scheduled,  is  sufficiently  mature,  and  is  satisfying  user  requirements. 

FYPT.ANATTON:  These  processes  institutionalize  the  software  development  process.  All  five  of  the 

processes  must  be  in  place  to  ensure  properly  configured  software  for  OT&E. 

RFFFRFNCE  QUESTION: 

_ SPM(IM)-012  The  contractor's  system  software  tool  environment  is  adequate. 

4.1.1.  AUDITS^IEVIEWS 

FACTOR:  The  software  development  audit  and  review  processes  indicate  development  of  a  maturing 
product. 

EXPLANATION:  Software  development  audits  and  reviews  are  held  to  assess  the  status  of  the  work 
in  progress  and  to  assess  the  risk  associated  with  proceeding  into  the  next  phase  of  development. 
Audits  and  reviews  can  be  used  to  indicate  the  product’s  maturity  by  evaluatmg  them  to  discover:  if 
the  scheduled  audit  or  review  has  taken  place,  if  action  items  identified  during  the  audit  or  review 
have  been  further  addressed/resolved,  and  if  the  audit  or  review  has  been  approved.  If  an  audit  or 
review  is  scheduled  and  does  not  take  place,  this  is  an  indication  the  product  has  not  processed 
sufficiently  in  its  evolution.  Further,  if  there  are  unresolved  action  items  or  an  audit  or  review  has 
been  disapproved,  this  may  be  an  indication  of  an  immature  product. 

The  audits  and  reviews  you  could  assess  to  determine  process  sufficiency  are:  System  Requirements 
Review  (SRR),  System  Design  Review  (SDR),  Software  Specification  Reviews  (SSR),  PDR,  CDR,  Te^ 
Readiness  Reviews  (TRR),  Functional  Configuration  Audit  (FCA),  and  Physical  Configination  Audit 
(PCA).  Note  the  focus  of  ffie  evaluation  of  these  audits  and  reviews  is  not  to  assess  their  quality  but 
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to  determine  whether  they  are  taking  place  as  scheduled  and  are  identifying  and  mdicalmg 
resolution  of  problem  areas.  See  MIL-STD-1521B,  appendices  A  through  H,  for  an  expansion  of  what 
each  of  the  audits  and  reviews  is  intended  to  cover. 

ppgpnxrgF  TMfiTPTTfyrTONS:  If  a  scheduled  audit  or  review  has  not  taken  place,  rate  the  fartor 
red.  In  addition,  action  items  that  have  been  raised  but  never  further  addressed  or  reviews  ^ 
that  have  been  disapproved  should  be  considered  possible  reasons  for  a  yellow  or  red  ratmg  depend¬ 
ing  upon  the  severity  of  the  issue. 


RF.1?F.RENCE  QTTESTIONS: 

SPM(PL)-004  Computer  resources  have  been  adequately  addressed  as  major  considerations  at 
procurement  reviews,  audits,  and  management  evaluations. 

__SPM(DM)-010  The  profile  of  unresolved  software  review  action  items  is  reasonable. 

4.1^.  CONFIGURATION  MANAGEMENT 


FACTOR:  The  SCM  processes  are  adequate. 


STTBFACTORS: 

_ ^identification  (4.1.2.1)  -  Acceptable  procedures  are  used  for  identifying  software  baselines. 

CONTROL  (4.1.2.2)  -  Acceptable  procedures  to  control  changes  to  the  baseline  are  practiced. 

AUDIT  (4. 1.2.3)  -  Acceptable  procedures  are  used  to  audit  development  progress  for  compliance 
to  SCM  requirements. 


ACCOUNTING  (4.1.2.4)  -  Acceptable  procedures  for  reporting  configuration  management  events 
are  practiced. 


FYPT.ANATION:  SCM  nrocesses  should  be  integrated  into  the  software  development  effort,  and  an 
S^Sd  exist  I  document  a  change  and  the  rationale  for  ihe  ch^ge.  S^M  ^sse^ 
should  not  unnecessarily  burden  or  slow  down  the  development  effort  and  should  be  automated 
the  maximum  extent  possible. 


■n.  assess  the  SCM  proeess,  visit  the  eentractofs  SCM  erganisation  and  Jis^ 

software  configuratim  manager.  If  this  is  not  possible,  an  alternative  is  the  SPO  SCM  organization. 


REFERENCE  QUESTIONS: 

SPM(IM)-011  The  contractor's  software  configuration  management  support  tool  environment 

is  adequate.  . 

SPM(TS)-019  The  configuration  management  of  the  software  test  process  is  ad^uate. 

_ SCM(CC)-014  The  contractor's  automated  support  tools  for  configuration  control  of  basehnes 

gnd  internal  development  identifications  is  adequate. 


4.I.2.I.  IDENTIFICATION 

FACTOR:  Acceptable  procedures  are  used  for  identifying  software  baselines. 

EYPT  ANATION'  The  nrocedures  need  to  be  structured  to  accomplish  at  least  two  fiinctions.  First, 
S^Mrf  to  bTztrucliired  in  a  manner  that  permits  identification  of  both  maior  and  ^or 
and  nnmbers  of  copies.  Second,  tbe  procedures  must  comcide  with  the  development  of  the  configur 
tion  status  accounting  tree. 
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Two  types  of  documents  need  to  be  reviewed  to  rate  this  factor.  First,  the  software  configuration  tree 
needs  to  be  reviewed  for  adequacy,  completeness,  and  currency.  Second,  comp^e  the  tree  structure 
and  numbering  to  several  software  configuration  items.  Use  the  results  of  this  comparison  to  rate 

the  factor. 

T?P!.c!PON.ciR  INSTRUCTIONS:  If  the  SCM  identification  procedures  are  established  but  are  not 
being  adhered  to  and  a  corrective  action  plan  exists,  rate  yellow  or  green.  K  SCM  identification 
procedures  do  not  exist  and  software  module  partitions  exist,  rate  red,  as  formal  SCM  cannot  be 
maintained. 

REFERENCE  QUESTIONS: 

_ SCM(ID)-003  The  procurement  activity  identification  of  the  software  configuration  baselines  is 

adequate. 

_ SCM(ID)-007  The  identifier  characteristics  for  software  configuration  item  names  are 

adequate.  i  -j 

_ ^SCM(ID)-010  Contractor's  deliverable  configuration  items  are  named  to  adequately  identify 

multiple  versions  and  variations. 

__SCM(ID)-011  Contractor's  identification  procedures  are  structured  to  permit  easy  addition, 
deletion,  or  modification  of  configured  items  at  any  hierarchical  level. 


CONTROL 

FACTOR:  Acceptable  procedures  to  control  changes  to  the  baseline  are  practiced. 

EyPT.ANATTON:  SCM  control  procedures  need  to  remain  flexible  early  in  the  software  development 
to  accommodate  rbangiug  user  requirements  and/or  changes  to  the  sofl:ware  modules  and  hardware. 
This  can  be  established  by  a  review  of  overall  Software  Change  Request  (SCR)  processing  times. 
Formal  control  early  in  the  development  will  adversely  affect  the  development  and  schedule.  As  the 
modules  approach  FCA/PCA,  the  SCM  control  procedures  need  to  be  more  formal. 

The  Configuration  Control  Board  (CCB)  should  be  functioning,  have  the  appropriate  agency  repre¬ 
sentatives  in  attendance,  and  have  sufficient  authority  to  implement  the  control  policies.  CCB 
approval  should  be  required  to  sanction  a  baseline.  To  determine  these,  review  the  CCB  meeting 
minutes.  Accurate  up-to-date  status,  tracking  of  changes  that  is  timely  and  responsive,  and  positive 
control  of  each  SCR/ECP  are  attributes  of  effective  SCM  control  procedures. 

]f?F..ciPONSE  INSTRUCTIONS:  If  the  procedures  are  established  and  adequate  but  not  totally 
adhered  to  and  a  corrective  action  plan  exists,  rate  the  response  yellow  or  green  as  appropriate.  If 
neither  is  true,  rate  the  response  red. 


REFERENCE  QUESTION: 


_ ^SCM(CC)-011  The  contractor's  configuration  control  board  has  an  adequate  interface  with  the 

procurement  activity  configuration  control  board. 


AUDIT 

FACTOR:  Acceptable  procedures  are  used  to  audit  development  progress  for  compliance  to  SCM 
requirements. 

EYPT.ANATION:  The  SCM  audit  procedimes  should  be  clearly  documented.  Configuration  identifi¬ 
cation  tools  can  indicate  which  elements  of  the  configuration  identification  have  been  changed  as  a 
/.ntifinnntinn  of  the  incorporated  changes.  Automated  tools  greatly  facilitate  the  efficiency  and  accu¬ 
racy  of  the  audit/review  activity. 
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5:  If  a  beiseline  is  established  without  an  audit,  or  audit  procedures  are 


not  being  followed,  rate  red. 


_ SCM(AR)-005  The  software  product  acceptance  requirements  are  adequate. 

_ SCM(AR)-006  The  contractor's  internal  configuration  audit/review  process  facihtates  the 

development  ofhigh  quality  production,  software. 

SCM(AR)-008  The  contractor's  configuration  management  tool  support  facihtates  e 
audit/review  of  the  process  by  which  changes  are  mcorporated  mto 
configuration  identifications. 

4.1^.4.  ACCOUrmNG 

FAnTOR:  Acceptable  procedures  for  reporting  configuration  management  events  are  practiced. 

FYPT.ANATION:  Confieuration  management  accounting  deals  with  the  extent  and  quality  to  w^h 
sS^ordkeeping  is  performed.  SCM  procedures  to  record  and  report  sanctioned  basehnes  ^Ps, 
SCRs,  and  software  trouble  reports  need  to  be  established  and  used  at  the 

development.  ECP  and  SCR  records  firom  change  inception  to  compleUon  shoi^  be  coinplete  to 
provide  an  audit  trail.  Software  trouble  reports  should  be  associated  with  the  software  module  and 

track  all  actions  taken. 

RFSPONfiF.  TNSTRTTf!TTQNS:  If  a  small  number  of  SCRs  do  not  have  adequate  SCM  accounting 
records,  rate  the  response  yellow  or  green  as  appropriate.  If  a  majority  do  not  have  adequate  re- 
cords,  rate  the  response  red. 


_ ^SCM(ID)-015  The  contractor's  software  change  control  forms  are  adequate.^ 

_ SCM(CC)-002  The  procurement  activity  has  implemented  adequate  software  coimguranon 

management,  based  upon  regulations,  to  control  the  fiincbonal  and  physical 

characteristics  of  all  CSCIs.  ,  ,-n  t  j  m  »=  tt 

_ ^SCM(CC)-005  The  procurement  configuration  control  procedures  for  the  Class  I  and  Olas 

changes  (or  equivalent  categories)  are  adequate. 

_ ^SCM(SA)-004  The  procurement  activity  configuration  status  accounting  procedures  are 

adequate. 

_ SCM(SA)-005  The  contractor's  internal  configuration  status  accountmg  procedures  ar 

adequate.  ^  ^  ^  xi  „ 

_ SCM(SA)-008  The  contractor's  automated  support  tools  for  configuration  status  accountmg  ot 

bflgAlinftg  and  internal  development  identifications  are  adequate. 

_ SCM(SA)-009  The  contractor's  software  configuration  status  accounting  forms  are  adequate. 

4.1^.  QUAUTY  ASSURANCE 

FACTOR:  The  software  quality  assurance  policies  and  procedures  are  effective. 

FYPT  .ANATION:  Assess  quality  assurance  by  reviewing  the  following: 

(1)  Sufficient  technical  quality  and  integrity  guidelines  to  monitor  product  quality  are 

established.  ,  i  4.  j 

(2)  Operational  engineering  and  product  impacts  of  suggested  basehne  changes  are  evaluated 

before  they  are  made. 

(3)  Reviews  are  structured  (implemented)  to  checkpoint  the  quality  of  data  before  they  are 
placed  tinder  control. 
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(4)  A  set  of  planned  (implemented)  quality  inspections  are  in  place  to  assess  software  products 
and  process  quality. 

(5)  Procedures  are  in  place  to  provide  guidance  on  what  is  expected  in  development  testing. 

(6)  The  integrated  set  of  software  quality  test  levels  is  structured  to  provide  smooth  data  flow 
nnd  smooth  transition  of  responsibility. 

(7)  Independent  observations  of  software  product  and  process  quality  sre  planned 
(implemented). 

(8)  Software  management  metrics  are  tracked  and  used  for  their  intended  purpose. 

The  objective  of  the  software  quality  organization  is  to  provide  a  system  of  checks  and  balances  that 
identifies  potential  quality  problems  before  they  become  significant.  The  establishment  and  use  of 
software  quality  assurance  pohcies  and  procedures  contribute  to  the  contractors  abihfy  to  develop 
operational  software.  The  areas  of  software  quality  management  include  qu^ty  reviews,  independ¬ 
ent  quality  assurance,  independent  verification  and  validation,  and  test  integration  of  software 
testing. 

T.TFRfryriT.y.  APPI-THATION:  All  phases  of  the  software  development  effort.  During  the  early 
stages  of  software  development,  software  quahly  assurance  standards  and  procedures  should  be 
developed.  Planning  for  IV&V  should  begin  early,  starting  with  an  assumption  of  organic  software 
support  and  use  of  the  Air  Logistics  Center  (ALC)  as  the  IV&V  agent.  As  development  continues,  the 
software  quality  assurance  practices  should  be  monitored  to  ensure  they  are  applied  and  to  identify 
shortfalls  and  improve  the  policies. 

RPJ.ATPn  METRICS:  SEI  assessment  results. 

REFERENCE  QUESTIONS: 

_ ^SPM(PL)-015  Software  quality  assessment  procedures  have  been  adequately  defined  to  meet 

management  policies  and  appropriate  regulations,  conform  to  standards,  and 
meet  performance  and  quality  requirements  throughout  the  system  life  cycle. 

_ ^SPM(PL)-027  Planning  for  systematic,  quantitative,  and  objectively  reportable  software  tests 

has  been  adequate. 

4.1.4.  PROGRAM  MANAGEMENT 

FACTOR:  Contractor's  program  management  processes  in  place  to  determine  implementation 
methodologies,  assess  progress,  and  identify  shortfalls  are  adequate. 

SUBFACTOBS: 

RESOURCE  FUNCTIONS  (4.1.4.1)  -  The  contractor's  program  methodologies  that  project, 
structure,  and  monitor  resources  are  adequate  to  complete  the  software  development  and  develop¬ 
mental  testing. 

_ ^CONTROL  FUNCTIONS  (4.1.4.2)  -  The  contractor's  program  management  control  functions  are 

adequate  to  ensure  software  development  completion. 

TOrPT  ANATION:  Two  indicators  which  determine  contractor's  program  management  process  mati^- 
ify  are  resource  functions  and  control  functions.  The  management,  technical,  administrative,  quahty 
assurance,  and  configuration  management  structures  should  be  tailored  and  sized  to  meet  the 
specific  needs  of  the  program.  The  structures  should  be  organized  so  they  are  responsive  to  program 
needs. 

A  system  of  checks  and  balances  exists  between  program  management  and  configuration  manap- 
ment  nnd  between  program  management  and  quality  assurance.  Consequently,  the  configuration 
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management  and  quality  assurance  ftmctions  are  accomplished  even  if  it  slows  the  dewlopment 
effort.  The  management  structure  should,  therefore,  appropriately  allocate  authority  to  the 
configuration  management  and  quality  assurance  functions. 

REFERENCE  QUESTION: 

SPM(IM)-005  The  implementation  methodology  used  by  the  contractor's  is  adequate. 

4.1.4.1.  RESOURCE  FUNCTIONS 

FACTOR:  The  contractor's  program  methodologies  that  project,  structure,  and  monitor  resources  are 
adequatft  to  complete  the  software  development  and  developmental  testing. 

SUBFACTORS; 

PLANNING  (4. 1.4. 1.1)  -  Adequate  resources  are  planned  for  software  development  and  devel¬ 
opmental  testing. 

ORGANIZATION  (4.1.4.1.2)  -  The  software  program  management  organizational  structure 
supports  the  software  development  schedule. 

MONITORING  (4.1.4.1.3)  -  The  software  development  program  management  monitoring  func¬ 
tion  is  adequatft  to  assess  that  software  development  milestones  can  be  met. 

EYPT.ANATION:  Contractor  activity  is  required  to  provide  certain  management  information  on  the 
project  status  to  the  procurement  activity.  The  contractor's  software  project  management  support 
tool  environment  should  be  adequate.  Review  the  contractor's  use  of  support  tools  to  perform  data 
collection  and  processing.  The  contractor's  software  resource  support  methodologies  should  be  ade¬ 
quate  to  identify  shortfalls  and  make  adjustments. 

T7i?gpr>USF.  TN.c;TRTTCTTONS:  If  no  project  management  tools  are  being  used  but  the  necessaiy 
resource  management  information  is  being  identified,  rate  the  response  yellow  or  green  as  appropri¬ 
ate.  If  project  management  tools  and  information  are  not  being  utilized,  rate  the  response  red. 

REFERENCE  QUESTIONS: 

__SPM(OS)  008  The  contractor's  personnel  stafBng  has  been  adequate  throughout  the  software 

life  cycle  phases.  ^  ,  u  f 

SPM(OS)  009  The  ratio  of  experienced  contractor's  project  personnel  to  the  total  number  ot 
project  personnel  has  been  adequate  throughout  the  software  life  cycle  phases. 
SPM(OS)-010  The  number  of  contractor's  personnel  has  been  adequate  through  out  the 
software  life  cycle  phases. 

SPM(PL)-004  Computer  resoiu-ces  have  been  adequately  addressed  as  major  consider-actions 
at  procurement  reviews,  audits,  and  mansigement  evaluations. 

SPM(PL)-029  Tracking  ofcomputer  resource  utilization  has  been  adequately  planned. 

SPM(IM)-010  The  contractor's  software  project  management  support  tool  environment  is 
adequate. 

4.1.4.1.1.  PLANNING 

FACTOR:  Adequate  resources  are  planned  for  software  development  and  developmental  testing. 

EyPT.ANATION:  Adequate  software  resource  planning  and  detailed  projections-to-complete  permit 
the  software  manager  to  minimize  software  development  schedule  delays  and  cost  overruns.  Devel¬ 
opment  of  resource  plans  utilizing  automated  tools  permits  fi-equent  updates.  The  plans  and  projec- 
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tions  the  software  manager  develops  need  to  be  reviewed  for  adequacy.  A  review  for  adequacy 
includes  to  ensure  the  plans  will  ultimately  deliver  producte  that  meet  user  requirements. 

Comparison  of  the  planning  to  the  current  implementation  will  provide  information  on  the  adequacy 
of  resource  planning. 

The  software  manager’s  development  plan  should  reflect  the  developmental  timeline,  realistic  re¬ 
quirements,  and  adequate  resources  to  meet  the  requirements  and  contain  built  in  contingency  pads. 
The  resources  development  plan  should  include  a  monitoring  of  resources  consiuned  to  provide  a 
check  on  resource  effectiveness. 

TtF.SPONSF.  INSTRUCTIONS:  If  resource  planning  is  less  than  complete,  rate  the  response  yellow 
or  green  as  appropriate.  If  resource  planning  was  not  performed,  rate  the  response  red. 

RFFERFNCE  QUESTIONS: 

SPM(PL)-007  Margins  for  reserve  computer  resource  capacity  to  provide  for  later  product 
improvements  are  adequate. 

_ SPM(PL)-029  Tracking  of  computer  resource  utilization  has  been  adequately  planned. 

4.1.4.!^.  ORGANIZATION 

FACTOR:  The  contractor's  software  program  management  organizational  structure  supports  the 
software  development  schedule. 

EYPT.ANATTON:  One  key  to  developing  and  supporting  software  is  to  have  e3q)erienced  personnel, 
especially  in  the  key  leadership  positions.  Experience  with  the  subject  system,  similar  systems, 
technologically  similar  problems,  the  management  problems  of  sunilar  systems,  and  the  interfaces 
with  the  subject  system  results  in  better  managed,  higher  qualify  software  products.  Experienced 
personnel  also  shorten  the  learning  curve  for  less  experienced  personnel.  Therefore,  the  contractor's 
personnel  eiqperience  mix,  including  management  staff,  planned  for  application  to  the  project  should 
be  consistent  with  the  productivity  realities  of  the  project. 

RESPONSE  INSTRUCTIONS:  If  the  indicators  reveal  resoiirce  and/or  schedule  deficiencies,  rate 
the  response  green  or  yellow  as  appropriate.  If  no  contractor's  resource  or  schedule  management  is 
being  performed,  rate  the  response  red. 

REFERENCE  QUESTION: 

SPM(OS)-009  The  ratio  of  experienced  contractor's  project  personnel  to  the  total  number  of 
project  personnel  has  been  adequate  throughout  the  software  life  cycle  phases. 

4.1.4.!^.  MONITORING 

FACTOR:  The  contractor's  software  development  program  management  monitoring  function  is 
adequate  to  assess  the  software  development  milestones  can  be  met. 

EXPTANATION:  Contractor's  management  monitoring  of  the  software  development  process  and  the 
resources  used  should  be  continuous  throughout  development.  The  efficient  application  of  resources 
jjy  innrtngpfnpnt  can  be  achieved  only  by  continual  monitoring  of  resource  consumption  and  progress 
and  milestone  achievement  status. 

The  staffing  continuity  is  determined  by  the  rate  of  turnover  of  personnel  during  and  across  the 
lifecycle  phases.  If  the  same  personnel  (or  at  least  a  reasonable  ratio  of  the  same  personnel)  are  not 
available  from  phase  to  phase,  then  there  is  likely  to  be  a  perturbation  in  the  schedtile,  cost,  func- 
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tional  requirements,  and  quaUty  of  the  deUverable  products.  Turnover  of  key  personnel  should  be 
minimal  with  no  sharp  variations. 


RTCSPONST^  INSTRUCTIONS:  Interviews  with  the  contractor's  software  development  team  and 
project  management  are  necessary  to  assess  the  level  at  which  the  contractor  management  is  mom- 
toring  the  progress.  If  management  is  vmaware  of  developmental  status  to  the  pomt  of  bemg  un¬ 
aware  of  missed  milestones,  rate  the  response  red. 


To  assess  personnel  continuity: 

Red  50%  or  more  turnover  during  any  one  acquisition  phase  (Concept,  Demonstration, 
Developement) 

Yellow  20%  to  50%  turnover  during  any  one  acquisition  phase 
Green  0%  to  20%  turnover  during  any  one  acquisition  phase 


T^FtTiATF^  MP.TRTCS:  Software  Development  Personnel. 

The  software  development  personnel  metric  reflects  the  ciment  status  of  the  development  staff  to 
include  staff  turnover.  This  indicator  compares  the  following  staff  counts:  tot^  placed,  plann 
experienced,  total  actual,  actual  experienced,  unplanned  losses  (total,  mexperienced  ^d  e^en- 
enced).  See  AFP  800-48  for  more  information  on  the  calculation  and  mterpretation  of  the  software 

development  personnel  metric. 


REFERENCE  QUESTIONS: 

_ SPM(OS)-008  The  contractor's  personnel  staffing  has  had  continuity  throughout  the 

software  life  cycle  phases.  ,  .  a. 

_ ^SPM(PL)-008  Acceptable  techniques  have  been  used  to  estimate  and  momtor  software  cos 

throughout  the  system  life  cycle. 


4.1.4^.  CONTROL  FUNCTIONS 

FACTOR:  The  contractor's  program  management  control  functions  are  adequate  to  ensure  software 
development  completion. 


SUBFACTORS: 

CORPORATE  POLICIES  (4.1.4.2.1)  -  The  contractors  corporate  software  development  policies 
are  adequate  to  support  the  software  development  schedxile. 

_ ^SUBCONTRACTOR  MANAGEMENT  (4.1.4.2.2)  -  The  subcontractorls)  software  development 

effort  is  adequate. 


_ COMMUNICATION  INTERFACES  (4.1.4.2.3)  -  The  communication  interfaces  are  adequate. 

EXPT.ANATION:  Contractor's  management  control  procedures  should  be  integrated  into  th^ro- 
gram  environment  and  tailored  to  the  specific  needs  of  the  program.  Management  control  ^<^0“ 
diould  be  automated  to  the  extent  possible.  The  scope  of  the  management  environment  should  be 
defined  and  understood,  and  there  should  be  a  contingency  plan  to  address  the  completion  of  tasks 

on  the  critical  path. 


Both  software  project  management  and  overall  program  management  control  is  needed  to  complete 
the  software  development.  At  the  management  level,  control  consists  of  afl  basic  management  func¬ 
tions  necessary  to  sustain  the  development  schedule. 
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The  contractor's  management  structure,  management  control  techniques,  and  the  management 
environment  should  be  reviewed  and,  if  needed,  tailored  to  the  specific  program.  A  chronological 
review  of  the  developmental  effort,  first  with  the  SPO  software  project  officer  and  then  with  the 
contractor  program  and  software  project  managers,  will  provide  the  information  required  to  rate  this 
factor. 

4.1.4,2.1.  CORPORATE  POUCIES 

FACTOR:  The  contractor's  corporate  software  development  policies  are  adequate  to  support  the 
software  development  schedule. 

TCXPTANATIQN:  The  contractor's  corporate  and  local  policies  should  agree  with  program  needs,  and 
the  contractor's  total  quality  management  policy  should  meet  program  needs.  The  contactor's  early 
policy  decisions  can  have  major  impact  on  the  software  development.  You  should  consider  several  of 
the  following  policies  that  may  impact  the  program:  software  make  or  buy  policy,  developmental 
hierarchy  of  testing  policy,  software  or  firmware  designation  policy,  and  subcontracting  of  IV&V 
policy. 

Review  each  of  the  contractor's  corporate  policies  with  the  contractor  s  software  project  manager. 
Any  policy  that  could  adversely  impact  OT&E  should  also  be  discussed  with  the  program  office. 

RELATED  METRICS:  SEI  Assessment  Results. 

4.1.4.2^.  SUBCONTRACTOR  MANAGEMENT 

FACTOR:  The  subcontractor's  software  development  effort  is  adequate. 

FYPT.ANATTON:  The  subcontractor's  software  development  schedule  should  have  the  same  features 
as  the  prime  contractor's  schedule.  In  addition,  the  prime  contractor's  and  subcontractor  s  schedule, 
standards,  methods,  SCM,  and  quality  assurance  policies  should  mesh. 

RF.SPONRF  INSTRUCTIONS:  If  there  are  no  subcontractors  with  software  development  responsi¬ 
bilities,  disregard  this  factor. 

REFERENCE  QUESTIONS: 

__SPM(DM)-015  The  contractor's  monitor  of  the  subcontractor  software  design  process  has  been 
adequate. 

_ ^SPM(IM)-007  The  contractor's  monitor  of  the  subcontractor  software  implementation  process 

has  been  adequate. 

4.1.4^.3.  COMMUNICA'nON  INTERFACES 

FACTOR:  The  communication  interfaces  are  adequate. 

FyPT.ANATTON:  The  software  development  program  management  function  should  firequently  com¬ 
municate  development  status  to  the  software  development  team,  subcontractors,  testers,  and  the 
program  office. 

The  internal  interfaces  within  the  contractor's/subcontractor  organization  elements  should  be  ade¬ 
quate.  Characteristics  to  be  assessed  include  proper  decision  process  information  flow,  effectiveness 
of  information  flow,  and  adherence  to  regulations  and  guidelines  for  interface  responmbility. 
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Informal  interviews  will  reveal  tiie  extent  to  which  communication  is  occurring.  If  communication 

win  be  apparent.  Identiflcatien  of  the  cauae  ia  beyond  the  of  ta 

SOA. 

T?F.RP0NSF;  TTCSTHTTOTIONS:  Rate  the  response  red  if  communication  problems  put  software 
development,  DT&E,  or  OT&E  at  risk. 


SPM(OS)-O15 

SPM(IM)-002 

SPM(PI)-012 

__SPM(PI)-013 

SPM(P1)-014 

_ ^SCM(AR)-008 


Internal  interfaces  among  contractor's  organization  elements  have  been  adequate 
The  procurement  test  organization  interface  with  the  contractors  is  adequate 
enough  to  ensure  success  of  the  system  integration  tests. 

The  IV&V  agency  external  interfaces  are  adequate. 

The  software  configuration  management  interfaces  among  all  activities  manage¬ 
ment  components  for  the  subject  system  are  adequate.  ^  . 

The  software  quality  assurance  management  interfaces  among  aU  activities 

management  components  for  the  subject  system  are  adequate. 

The  contractor's  configuration  management  tool  support  facihtates  toe 
audit/review  of  toe  process  by  which  changes  are  incorporated  into  configuration 

identifications. 


4.1.5.  TESTING 

PAflTOR:  The  testing  processes  are  adequate  to  ensure  software  is  being  tested  as  scheduled,  is 
sufficiently  mature,  and  is  satisfying  user  requirements. 

RYPT  ANATION-  A  mature  testing  process  is  one  that  ensures  aU  toe  software  is  tested,  toe  tests 
^i^^y  pmSems,  as  teatog  continues  the  software  faUare  rates  d^as^  and  ^ 
requirem^  are  being  met.  It  should  be  apparent  from  the  description  of 
du^d  and  their  results  whether  or  not  previous  goals  have  been  met  and  test 
satisfied  Vague  references  to  "successful  software  results"  or  no  problems  with  toe  softw^e 
should  not  be  Sieptable.  In  order  to  evaluate  toe  progress  of  software  ^ 

expUcit  reference  to  a  systematic,  scientificaUy  sound  approach  to  carrymg  out  toe  test,  toe  relate 
sffip  between  toe  systematic  test  approach  and  toe  test  objectives  for  toe  current  phase,  toe  results  of 
toe  test,  and  toe  plans  for  resolution  of  errors. 

You  should  review  toe  STP  and  toe  STD  documents  to  determine  what  testog  is 

STP  will  describe  the  nature  and  extent  of  toe  testing  to  be  performed.  Tfie  testing  “ 

toese  documents  should  be  comprehensive  (testing  all  processing  flows,  ^decision  points,  md^ 

functions)  to  toe  maximum  extent  allowed  by  program  resources.  In  addition,  toe  mitial  ^stog 

should  be  stand  alone,  in  toe  modules  are  tested  independent  from  related  ®°‘^®- 

should  be  followed  by  procedures  that  "build"  to  test  of  grouped  modffies,  imffi 

integration  test  of  toe  software  is  planned.  The  test  planning  should  indicate  tests  for  both  vahd  and 

invalid  data,  as  well  as  stress  test  and  security  tests  (where  applicable). 

The  STP  will  have  a  test  schedule  you  can  use  to  match  against  current  stoedules  to  deterge  if  the 
testing  required  by  toe  documents  is  being  accomplished.  A  review  of  Software  Development  Pile 
required  by  toe  SOW)  may  also  provide  an  indication  of  toe  level  and  amount  of  testing  bemg  per- 

formed  on  the  software. 

To  produce  a  valid  test,  validated  test  data  must  be  available.  Before  testing  ^® 

should  be  reviewed  to  ensure  test  completeness,  reproducibility,  integrity,  and  predictab  ty. 
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REFERENCE  QUESTIONS: 

_ SPM(PL)-026  The  principles  and  methodologies  provided  in  the  regiJations  have  been 

appropriately  incorporated  into  the  software  test  and  evaluation  plans. 

_ SPM(PL)-027  Planning  for  systematic,  quantitative,  and  objectively  reportable  software  tests 

has  been  adequate. 

_ SPM(DM)-008  The  number  of  software  requirements  which  cannot  be  tested  are  minimal. 

_ SPM(IM)-013  The  contractor's  application  test  software  tool  environment  is  adequate. 

_ SPM(TS)-002  The  software  test  process  for  DT&E  has  followed  the  guidelines  in  the  TEMP. 

_ SPM(TS)-009  The  software  test  approach  and  methodologies  employed  are  clearly  described  in 

the  software  test  documentation  and  appear  to  be  effective. 

_ SPM(TS)-010  The  software  features  to  be  tested  and  not  to  be  tested  are  clearly  described  in  the 

software  test  documentation. 

4^.  PRODUCTS 

FACTOR:  Development  and  maturity  of  the  software  products  is  adequate  to  ensure  properly  con¬ 
figured  operational  software. 

SUBFACTORS: 

_ ^DOCUMENTS  (4.2.1)  -  The  software  documentation  set  has  progressed  to  the  appropriate  stage 

of  completion. 

_ SOFTWARE  (4.2.2)  -  The  software  code  is  acceptable  for  the  current  stage  of  development. 

EXPLANATION:  To  assess  the  software  development  products,  review  the  software  documentation 
and  source  code  for  problem  areas.  Assess  problem  areas  to  determine  their  impact. 

4J2.1.  DOCUMENTS 

FACTOR:  The  software  documentation  set  has  progressed  to  the  appropriate  stage  of  completion. 
SUBFACTORS: 

SDP  (4.2.1.1)  -  The  content  of  the  SDP  is  acceptable  for  the  current  stage  of  development. 

_ SSDD  (4.2. 1.2)  -  The  content  of  the  SSDD  is  acceptable  for  the  current  stage  of  development. 

INTERFACE  DESIGN  DOCUMENT  (IDD)  (4.2.1.3)  -  The  content  of  the  IDD  is  acceptable  for 
the  current  stage  of  development. 

_ SQPP  (4.2. 1.4)  -  The  content  of  the  SQPP  is  acceptable  for  the  ciurent  stage  of  development. 

_ SCMP  (4.2.1.5)  -  The  content  of  the  SCMP  is  acceptable  for  the  current  stage  of  development. 

SDP  (4.2. 1.6)  -  The  content  of  the  SDD  is  acceptable  for  the  current  stage  of  development. 

STP  (4.2.1.7)  -  The  content  of  the  STP  is  acceptable  for  the  current  stage  of  development. 

_ SOFTWARE  REQUIREMENTS  SPECIFICATION  (SRS)  (4.2.1.8)  -  The  content  of  the  SRS  is 

acceptable  for  the  current  stage  of  development. 

INTERFACE  REQUIREMENTS  SPECIFICATION  (IRS)  (4.2.1.9)  -  The  content  of  the  IRS  is 
acceptable  for  the  current  stage  of  development. 
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TCYPTANATION:  The  status  of  the  contractor’s  software  documents  can  be  reviewed  to  provide 
i^ighte  to  product  maturity.  The  focus  of  this  portion  is  on  whether  the  doc^ents  are  available 
(when  they  should  be  and  in  the  appropriate  form  (e.g.,  draft,  final)),  contam  the  appropriate  com¬ 
puter  resources  information,  and  are  updated  to  reflect  program  changes. 


REFERENCE  QUESTIOHS: 

_ SPM(DM)  -  016  The  design  specifications  for  the  software  products  contain  adequate 

information  to  implement  the  software  with  the  required  functionahty  and 

within  the  schedule  and  budget  requirements. 

SCM(ID)  -  017  The  documentation  which  collectively  identifies  the  content  of  a  configuration 
item  is  adequate. 


4^.1.1.  SDP 

FACTOR:  The  content  of  the  SDP  is  acceptable  for  the  current  stage  of  development. 

FyPT.ANATTON:  The  SDP  describes  the  contractor's  plans  for  conducting  softw^e  development. 
The  SDP  forms  the  comparison  baseline  for  most  of  the  contractor  documentation.  Ensure  e 
following  computer  resource  information  is  in  the  SDP. 

(1)  An  overview  of  the  software  project  organization  structure.  _ 

(2)  An  identification  of  the  number  of  personnel  necessary  to  complete  the  software 

development  effort.  .  _  .  . 

(3)  A  description  of  the  organizational  responsibilities  for  perfomung  the  software  engineermg 

activities. 

(4)  A  description  of  eadi  software  development  activity  of  the  project. 

(5)  The  impact  of  any  security  or  safety  requirements. 

(6)  The  proposed  corrective  action  process. 

(7)  The  softwEffe  tools  necessary  to  perform  the  software  engineering  activities. 

(8)  The  software  product  development  activities. 

(9)  Plans  for  formal  qualification  testing. 

(10)  A  description  of  the  software  product  evaluation  activities. 

(11)  A  description  of  the  software  configuration  management  activities. 

Significant  problems  with  the  schedule  presented  in  the  SDP  could  indicate  serious  development  or 
maturity  issues.  Ensure  the  schedule  indicates  the  following: 

(1)  Dependencies  between  tasks. 

(2)  The  critical  path  for  the  software  development  schedule. 

(3)  Schedule  milestones  as  measurable  events. 

(4)  Cost  estimates  are  updated  as  the  development  proceeds. 

(5)  Funding  is  available  to  support  the  estimated  cost  and  schedule. 

(6)  The  cost  and  schedule  status  reports  are  useful  management  tools  (e.g.,  at  the  appropriate 
level  of  detail,  accurate,  and  up  to  date). 

(7)  Time  for  software  development  activities. 

(8)  Adequate  time  to  address  action  items  fi^jm  reviews  (i.e.,  PDR).  ,  ,  . 

(9)  Adequate  time  to  accommodate  corrections  of  any  deficiencies  discovered  durmg 

testing  (e.g.,  unit  test,  integration  test,  DT&E). 

To  determine  all  activities  are  included  in  the  schedule,  review  the  WBS  to  identify  each  task.  The 
WBS  should  include  software  development  activities  such  as  major  reviews  and  milestones  and  ^ 
used  as  the  basis  for  schedule  and  cost  estimates.  The  WBS  should  be  updated  throughout  the 
program  to  reflect  significant  changes. 
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4J2.i:2.  SSDD 

FACTOR:  The  content  of  the  SSDD  is  acceptable  for  the  ciurent  stage  of  development. 

EXPLANATION :  The  SSDD  is  the  primary  document  for  the  system  design.  It  should  contain  the 
following  computer  resoxurces  information: 

-  The  description  of  each  CSCI  includes  a  summary  of  the  CSCI  purpose,  a  definition  of  the 
interfaces  external  to  the  CSCI,  and  a  matrix  of  the  system  requirements  allocated  to  the 
CSCI. 

-  The  software  related  interfaces  associated  with  each  CSCI  include:  interfaces  external  to  each 
CSCI,  CSCI  to  CSCI  interfaces,  and  CSCI  to  HWCI  interfaces. 

-  The  description  of  the  processing  resources  specifies  the  hardware,  programming,  design, 
coding,  and  utilization  characteristics  of  each  resource.  For  example,  the  processing  capacity 
(absolute  and  spare),  growth  capacities,  and  diagnostic  capabihties. 

-  The  requirements  traceability  matrix  provides  traceability  of  the  system  requirements 
allocated  to  the  CSCIs  back  to  the  requirements  of  the  system  specification. 

REFERENCE  QUESTIONS: 

See  AFOTECPAM  99-102,  volume  3. 

4:2.1.3.  IDD 

FACTOR:  The  content  of  the  IDD  is  acceptable  for  the  current  stage  of  development. 

EXPT .  ANATIQN :  The  IDD  specifies  the  detailed  design  for  one  or  more  interfaces  between  CSCIs 
and  other  configuration  items  or  critical  items  and  is  predicated  upon  the  interface  requirements 
specification  (IRS).  It  should  (1)  contain  interface  diagrams  and  a  complete  data  element  definition 
table  for  each  data  element  to  be  transmitted  across  the  interface,  (2)  specify  the  relative  priority  of 
each  interface  and  of  each  message  to  be  transmitted  across  each  interface,  and  (3)  describe  the 
technical  details  of  each  communications  protocol  associated  with  each  interface. 

The  data  element  definition  tables  for  each  interface  include  the  following  information  (as  applica¬ 
ble):  data  element  identifier,  data  element  description,  CSCI  that  is  tiie  data  element  source,  CSCI 
that  is  the  data  element  user,  data  element  unit  of  measure,  valid  range  of  values,  accuracy  require¬ 
ments,  calculation  (or  refi'esh)  fi'equency,  validity  checks,  data  type,  data  format,  and  data  priority. 

The  technical  details  of  each  communication  protocol  include  (as  applicable):  message  fragmentation 
and  resissembly,  message  formatting,  error  control  and  recovery  procedures,  synchronization,  flow 
control,  data  transfer  rate,  routing,  transmission  services,  status,  and  security. 

4^.1.4.  SQPP 

FACTOR:  The  content  of  the  SQPP  is  acceptable  for  the  current  stage  of  development. 

EXPI., ANATTON :  The  SQPP  documents  the  contractor's  software  quality  program.  The  objective  of 
the  software  quality  program  is  to  improve  the  quality  of  the  deliverable  products,  the  nondeliverable 
products,  and  the  software  development  processes.  The  software  quality  program  includes  planning, 
assessment,  reporting,  and  follow-up  activities.  The  SQPP  should  describe  the  software  quality 
environment,  organizational  structure,  procedures  and  tools,  and  program  activities. 
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The  SQPP  describes  the  products  and  processes  (and  tools)  of  the  software  quality  program  activities. 
The  software  quality  program  activities  include  assessments  of  the  following:  documentation, 
storage  and  shipping  of  deliverable  software,  software  development  libraries,  software  development 
files,  software  change  requests,  software  testing,  software  installation  and  checkout,  nondeliverable 
software,  and  subcontractor  developed  software. 

4.2.1.5.  SCMP 

FACTOR:  The  content  of  the  SCMP  is  acceptable  for  the  current  stage  of  development. 

F.XPLANATION :  The  SCMP  identifies  the  responsibilities  of  the  software  configuration  manage¬ 
ment  (SCM)  organization  and  describes  the  policies  and  procedures  of  that  organization.  It  should 
define  the  allocation  of  CM  resources,  specify  the  planned  (or  existing)  CM  policies  and  procedures, 
identify  the  planned  (or  existing)  CM  tools,  and  establish  the  function  and  procedures  associated 
with  the  CCB. 

The  procedures  for  controlling  software  problems  and  changes  should  be  addressed  in  the  SCMP. 
These  procedures  should  include  a  description  of  the  software  configuration  control  flow  and  audit 
mechanisms,  a  detailed  description  of  the  software  problem  change  report,  and  the  responsibihties 
and  procedures  of  the  software  change  authority,  the  CCB. 

4.2.1.6.  SDD 

FACTOR:  The  content  of  the  SDD  is  acceptable  for  the  current  stage  of  development. 

EXPLAN  ATI  ON :  A  separate  SDD  is  written  for  each  CSCI.  Each  SDD  should  describe  the  internal 
organization  of  the  CSCI,  the  CSUs  of  each  CSC,  and  the  global  data  elements  of  the  CSCI. 

The  description  of  the  internal  organization  includes  summarization  of  the  CSCs  and  sublevel  CSCs, 
the  relationships  between  CSCs,  memory  allocation  and  processing  time,  definition  of  the  system 
requirements  allocated  to  each  CSC,  description  of  the  preliminary  design  in  terms  of  execution 
control  and  data  flow,  derived  requirements,  and  design  constraints. 

The  description  of  the  CSUs  of  each  CSC  includes  sinnmarization  of  the  CSUs,  relationship  between 
CSUs,  and  statement  of  the  design  requirements  for  each  CSU.  Examples  of  CSU  design  require¬ 
ments  include  I/O  data  elements,  algorithms,  and  error  handling. 

The  SDD  should  be  consistent  with  the  information  in  the  SRS. 

REFERENCE  QUESTIONS: 

From  AFOTECPAM  99-102,  volume  3. 

_ P-20  The  documentation  adequately  describes  the  external  interfaces. 

D-29  Any  dynamic  allocation  of  resources  is  explained  in  the  documentation. 

_ ^D-32  Storage  requirements  for  each  major  function  of  the  program  are  adequately 

described  in  the  documentation. 

_ ^D-59  It  is  easy  to  trace  the  progreun  control  flow  at  all  system  levels. 

4^.1.7.  STP 

FACTOR:  The  content  of  the  STP  is  acceptable  for  the  current  stage  of  development. 

EXPLANATION :  The  STP  describes  the  formal  qualification  test  plans  for  one  or  more  CSCIs.  The 
STP  also  identifies  the  software  test  environment  resoiirces  required  for  formal  qualification  testing. 
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It  should  describe  the  plan  for  implementing  and  controlling  the  resources  necessary  to  perform 
formal  qualification  testing,  describe  the  contractor's  plans  for  controlling  and  maintaining  each  item 
in  the  test  environment,  describe  the  formal  qualification  test  for  each  CSCI,  contain  (or  reference) 
the  formal  qualification  test  schedule,  and  describe  the  data  reduction  and  analysis  procedures. 

Software  test  environment  resources  include  software  items  needed  to  perform  formal  qualification 
testing  such  as  operating  systems,  compilers,  code  auditors,  dynamic  path  analyzers,  test  drivers, 
preprocessors,  test  data  generators,  and  post  processors. 

The  STP  contains  contractor  plans  for  the  test  environment  to  include  installing  and  testing  each 
item  prior  to  its  use. 

The  formal  qualification  test  requirements  for  each  CSCI  should  address  the  foUovring  points: 

( 1)  General  test  requirements  (e.g.,  error  detection,  error  recovery,  boundary  conditions). 

(2)  Test  classes  (e.g.,  stress,  timing,  erroneous  input,  maximum  capacity  tests). 

(3)  Test  levels  (e.g.,  CSC-CSC,  CSCI-CSCI,  CSCI-HWCI,  system  level). 

(4)  Test  definitions  (e.g.,  qualification  method,  requirements  cross-references,  assumptions). 

(5)  Test  schedule. 

REFERENCE  QUESTIONS: 

From  AFOTECPAM  99-102,  volume  3  and  volume  2. 

_ ^D-48  The  program  test  plan  is  adequately  described  in  the  documentation. 

_ ^D-49  a  useful  set  of  test  procedures  for  high  levels  of  program  testing  is  contained  in  the 

documentation.  •  j  •  4.1, 

_ ^D-50  a  useful  set  of  test  procedures  for  low  levels  of  program  testing  is  contamed  m  the 

documentation. 

_ ^D-51  The  limitations  and  incompleteness  of  the  test  procedures  are  described  in  the  docu¬ 
mentation. 

_ ^D-52  The  sample  test  data  are  adequately  described  in  the  documentation. 

_ ^D-53  Program  support  tools  that  aid  in  testing  the  program  are  adequately  doc^ented. 

_ ^D-54  The  documentation  describes  software  test  probes  that  aid  in  identifying  processing 

performance. 

_ SPM(TS)-011  The  traceability  software  features  tested/not  tested  to  the  software  functional 

requirements  is  described  in  the  software  test  documentation. 

_ SPM(TS)-013  The  software  test  criteria  used  to  determine  whether  each  test  has  passed  or 

failed  are  clearly  specified  in  the  software  test  documentation. 

_ SPM(TS)-014  The  personnel  groups  responsible  for  the  software  tests  are  adequately  identified 

in  the  software  test  documentation. 

_ SPM(TS)-015  The  high  risk  assvunptions  of  the  software  testing  approach  along  vrith 

contingency  plans  for  each  such  assumption  is  adequately  described  in  the 
software  test  docinnentation. 

_ SPM(TS)-016  The  schedule  for  software  test  milestones  is  adequately  specified  in  the  software 

test  documentation. 

_ SPM(TS)-017  Software  testing  is  adequately  prioritized  in  the  software  test  approach  according 

to  mission  criticality  concerns. 

_ SPM(TS)-018  The  software  test  environment  is  adequately  identified  in  the  software  test 

documentation  and  is  adequate  for  accomphshing  the  required  testing. 

4^.1^.  SBS 

FACTOR:  The  content  of  the  SRS  is  acceptable  for  the  current  stage  of  development. 
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EXPLANATION:  Ensure  the  SRS  specifies: 

-  The  engineering  and  qualification  requirements  for  a  CSCI. 

-  The  engineering  requirements  necessary  to  ensure  proper  development  of  the  CSCI,  external 
interfaces  of  the  CSCI 

-  All  of  the  capability  requirements  that  the  CSCI  must  satisfy 

-  Interfaces  between  the  capabilities  identified  in  the  previous  step 

-  CSCI  data  element  requirements 

-  Requirements  for  adapting  the  CSCI  to  site-unique  conditions  and  to  changes  in  the  system 
environment 

-  Sizing  and  timing  requirements 

-  Safety  and  security  requirements 

-  The  applicable  human  factors  engineering  requirements  for  the  CSCI. 

4^.1.9.  ms 

FACTOR:  The  content  of  the  IRS  is  acceptable  for  the  current  stage  of  development. 

EYPT.ANATTON:  The  IRS  specifies  the  requirements  for  one  or  more  interfaces  between  a  partic\il» 
CSCI  and  other  configuration  items  or  critical  items  in  the  system  of  which  it  is  a  part.  An  IRS  is 
used  to  provide  interface  requirements  in  a  separate  document  in  the  following  circumstances:  (1) 
there  are  many  interfaces,  (2)  there  are  many  development  groups  involved  in  conmn^cating  the 
requirements,  (3)  the  complexity  of  the  interface  is  such  that  a  separate  IRS  will  simplify  communi¬ 
cations,  or  (4)  two  or  more  contractors  are  developing  the  requirements.  When  ^  IRS  is  used  to 
specify  requirements  for  interfaces,  the  detailed  design  of  the  interfaces  is  provided  in  an  IDD. 

Interface  blo(^  diagrams  depict  the  relationship  of  the  CSCI  to  the  otiier  HWCIs,  CSCIs,  or  critical 
items  in  the  system  for  which  interfaces  are  specified. 

The  IRS  specifies  the  methods,  techniques,  tools,  facilities,  and  acceptance  tolerance  limits  necessary 
to  establish  that  each  interface  satisfies  the  identified  requirements. 

4,2^.  SOFTWARE 

FACTOR:  The  software  code  is  acceptable  for  the  current  stage  of  development. 

SUBFACTOR: 

__RELIABILITY  (4.2.2.1)  -  The  rate  of  software  failures  will  not  impact  the  start  or  completion  of 
OT&E. 

_ ^SCHEDULES  (4.2.2.2)  -  The  software  is  being  developed  according  to  schedule. 

_ ^SPARE  CAPACITY  (4.2.2.3)  -  Margins  for  reserve  computer  resource  capacity  are  adequate. 

MAmTATTJARTT.TTY  (4.2.2.4)  -  The  software  and  associated  documentation  are  easily  modified. 

FYPT.ANATION:  The  status  of  the  software  can  be  reviewed  to  provide  insight  as  to  the  maturity  of 
the  products.  To  be  mature,  the  software  should  show  a  decreasing  failure  rate  trend,  be  on  schedule 
in  terms  of  development,  should  be  reaching  dosure  on  user  spare  capadty  requirements,  and  should 
be  sufficiently  maintainable. 

RESPONSE  INSTRUCTIONS:  If  any  of  the  above  are  unacceptable  and  there  is  no  plan,  you  should 
answer  red.  If  an  adequate  plan  exits,  answer  yellow. 
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RELIABILITY 

FAfiTQR:  The  rate  of  software  failures  will  not  impact  the  start  or  completion  of  OT&E. 

fypt.ANATION:  Reliability  is  one  of  the  measures  of  software  maturity  m  indication  of 
S^Sware  development  effort  will  support  the  start  of  OT&E.  ReUability  -  detoed 
extent  to  which  the  software  will  perform  its  intended  function  mthout  f^ure  withm  a  specified 
time  period.  In  the  software  failure  rate  is  decreasing,  the  reliabihty  will  be  mcreasmg,  and  the 
software  can  be  interpreted  as  becoming  more  mature. 

For  moot  appUcations.  aoceptablo  lovols  range  fipm  to  toe  errors  *0®^  fte  tek 
Unfortunately  an  OA  may  be  performed  long  before  dehvered  hnes  of  code  are  kno^.  For  the  lacJs 
£f^aton,L  evaluator  may  be  able  to  obtain  iMormation  t^t 
for  developed  (at  the  time  of  the  OA)  lines  of  code  and,  hence,  determine  the  trend  of  the  ra  . 

]^f;T  .ATFr>  METRICS:  Software  reliability  metrics  are  defined  in  AFOTECPAM  99-102,  volume  6. 

4.2.2.2.  SCHEDULES 

FAOTOR:  The  software  is  being  developed  according  to  schedule. 

pr5rPT.ANATION:  Examine  the  schedules  for  source  code  development  and  testing  and  ensime  ihe 
SSi^eloped  on  schedule,  or  if  it  is  behind  schedule,  ^re  are  confmgency  Pla-  ^  ^ 
any  code  development  problems  within  the  schedule.  If  a  software  produrt  s  devekpment  is  not 
meeting  its  development  schedule,  this  can  be  an  indicator  of  ^ature  sol^are. 
meeting  the  schedule  include:  more  errors  are  being  found  and  corrected  than  were  anticipated 
required  access  to  computer  hardware  was  not  being  met. 

ffcspomcsf  INSTRUCTIONS:  You  must  decide  whether  lack  of  software  progress  toward  meetog 

inadequate  aehedule  or  Uie  echedule  ie  aetually  eorreet  tat  ojer 

iesuesijroblems  have  suifeced  reUted  to  progress  toward  matorily.  For  “am^®’  f^h^^^^so^ 
control  lack  of  adequate  software  standards  and  practices,  assignment  of  unskilled  software  Person 
nel,  and  many  other  factors  could  cause  the  introduction  of  more  e^re  mto  ^ 

would  have  been  anticipated  when  the  schedule  was  built.  Your  best  approach  is  to  ^etemine  & 
conditions  or  assumptions  under  which  the  schedule  was  made  and  to  m^e  a  judgment  about  &e 
caure  of  the  problem  It  is  expected  problems  not  related  to  current  schedule 

imder  other  maturity  factors.  However,  if  this  is  not  the  case,  such  problems  could  be  reported  here. 
If  software  is  not  meeting  its  development  schedule,  rate  this  factor  yellow  or  red,  as  appropriate. 

4.2.2.3.  SPARE  CAPACITY 

FACTOR:  for  reserve  computer  resource  capacity  are  adequate. 

FYPT.ANATION:  Requirements  for  computer  margins  and  the  mtial  values  for 
are  e^hshed  by  the  using  command’s  ORD.  These  margins  then  evolve  throughout  &e  software 
development  effort.  Margins  should  be  established  for  memory,  external  starve,  task  ut^ataon, 
SrSX^^rma^parameto.  and  «o  fbrth.  The  margin  of  «aarva  »  ^  f” 

software  sup^rtabiUty,  since  changes  will  usually  require  consumption  of  some  of  the  reserve. 

FFT  ATFU  METRICS:  Computer  Resource  Utilization.  The  computer  resource_utiliz^on  metric 
teaX  resoi^ru^  and  reserve  capacity  available  for  each  computer  resource.  For  each  computer 
resource  three  subLdicators  should  be  tracked.  These  three  subindicatora  are  central  processmg 
unit  (CPU)  utilization  (planned  vs.  actual  percentage  of  CPU  execution  toe  coMumed),  memo^ 
utilization  (planned  vs.  actual  percentage  of  memory  consumed),  and  I/O  utilization  (planned  vs. 
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actual  percentage  of  I/O  resources  consumed).  See  AFP  800-48  for  more  information  on  the 
calculation  and  interpretation  of  the  computer  resource  utihzation  metric. 

4^^.4.  MAINTAINABILITY 

FACTOR:  The  software  and  associated  dooimentation  are  easily  modified. 

FYPT.ANATION:  Maintainability  is  the  effort  needed  to  change  software,  where  change  could  occur 
to  correct  errors,  add  system  capabilities,  delete  features,  or  modify  the  software  to  become  compat¬ 
ible  with  hardware  changes.  It  is  a  prerequisite  to  start  and  complete  OT&E.  Maintainable  software 
results  fi:om  the  use  of  a  soimd  development  process  to  build  a  quality  product. 

If  products  are  available,  perform  some  type  of  AFOTECPAM  99-102,  volume  3  evaluation.  If 
products  are  unavailable,  look  at  the  contractor’s  coding  standards  and  evaluate  whether  the 
standards  will  produce  products  that  will  score  well  on  a  volume  3  evaluation. 

RFT.ATFD  MFTRTnS:  There  are  several  software  metrics  that  provide  an  indication  of  the  main¬ 
tainability  of  the  software  product.  Simple  metrics  include  the  effort  expended  to  correct  errors  or 
the  amount  of  time  problem  reports  are  remaining  open.  The  results  of  an  AFOTECPAM  99-102, 
volume  3  evaluation  provide  metric  values  for  the  maintainability  of  the  documentation,  source  code, 
and  implementation.  Complexity  metrics  have  been  established  in  Automated  Software  Tools 
System  (ASETS)  and  can  be  used  to  aid  in  evaluating  this  factor. 

6.  REQUIREMENTS  TRACEABIUTy 

The  one  significant  computer  resource  issue  not  addressed  in  other  categories  is  requirements  trace- 
ability.  It  should  be  noted  that  "computer  resource  requirements,"  as  used  in  this  section,  includes 
not  only  specific  computer  resource  requirements,  but  also  requirements  (or  functions)  allocated  to 
software. 

FACTOR:  There  are  no  missing  computer  resource  requirements  that  would  adversely  impact  the 
system. 

SUBFACTORS: 

ORD/RCM  TO  SYSTEM  SPECIFICATION  TRACEABILITY  (5.1)  -  The  user's  computer  re¬ 
sources  requirements  are  traceable  finm  the  ORD/RCM  to  the  system  specification. 

__DEVELOPMENT  CONTRACTOR  TRACEABILITY  (5.2)  -  The  contractor  has  adequately  traced 
the  user's  computer  resources  requirements  firom  the  system  specification  through  the  development 
documents. 

6.1.  ORD/RCM  TO  SYSTEM  SPECIFICA'nON  TRACEABILITY 

FACTOR:  The  user's  computer  resources  requirements  are  traceable  from  the  ORD/RCM  to  the 
system  specification. 

FXPTJ^ATTON:  The  ORD/RCM  should  identify  computer  resources  requirements.  It  should 
address  all  mission  critical  and  support  computer  resources,  including  automated  test  equipment. 
The  fapabi1iti«^s  desired  for  integrated  computer  resources  support  ^ould  be  described.  It  should 
describe  any  unique  interface  requirements,  documentation  needs,  and  special  software  certifica¬ 
tions.  Each  of  the  requirements  in  the  ORD/RCM  should  be  contained  in  the  system  specification. 
Methods  are  available  that  document  relationships  between  requirements,  and  create  a  require¬ 
ments  traceability  matrix  to  assist  the  user  in  veriftdng  requirements  allocation.  This  process  can  be 
done  manually  or  by  using  automated  tools,  but  a  disciplined  method  is  needed.  The  output  of  the 
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method  should  provide  some  form  of  a  requirements  traceabUity  matrix.  You  should  use  this  matrix 
to  determine  each  computer  resources  requirement  in  the  ORD/RCM  is  contained  in  the  system 
specification. 

In  to  the  specific  computer  resources  requirements,  many  of  the  user's  operational  effective¬ 

ness  and  operational  suitability  performance  requirements  identified  in  the  ORD/RCM  may  be 
allocable,  in  whole  or  in  part,  to  software.  The  system  specification  must  contain  each  of  these 
ORD/RCM  requirements. 

6^  DEVELOPMENT  CONTRACTOR  TRACEABnJTY 

FACTOR:  The  contractor  has  adequately  traced  the  user's  computer  resources  requirements  from 
the  system  specification  through  the  development  documents. 

F.yPT . ANATION:  Requirements  in  the  system  specification  are  allocated  to  lower  tier  specifications 
and  development  documents.  Each  requirement  in  the  system  specification  must  be  traceable 
through  all  the  oilier  development  documents.  Methods  are  available  that  document  relatioMhips 
between  requirements  and  create  a  requirements  traceability  matrix  to  assist  the  user  in  verifying 
requirements  allocation.  This  process  can  be  done  manually  or  by  using  automated  tools,  but  a 
disciplined  method  is  an  absolute  necessity  for  complex  systems.  You  should  determine  the  contrac¬ 
tor  has  a  disciplined  method. 

The  computer  resources  requirements  should  be  traceable  firom  the  system  specification  to  the  sys¬ 
tem  Hagign  document,  fi:nm  the  system  design  document  to  a  software  requirements  specification, 
and  tho"  fi-om  the  software  requirements  specification  to  the  software  design  document.  You  should 
select  a  representative  sample  (as  large  as  feasible)  of  computer  resources  requirements  in  tte  sys¬ 
tem  specification  and  trace  each  one  through  the  hierarchy  of  development  documents  usmg  the 
contractor's  requirements  traceability  method. 

REFERENCE  QUESTIONS: 

SPM(DM)-007  The  number  of  software  requirements  which  cannot  be  traced  to  an  end  item 
product  is  minimal. 

(Prom  AFOTECPAM  99-102,  volume  3) 

_ ^D-55  System  requirements  are  easily  traceable  to  the  actual  fimction/CSCI  that  implements  the 

requirement.  .,  ,  ,  .  x.  c 

_ D-56  a  function's  (or  CSCI's/CSC's)  description  is  easily  traceable  to  the  detailed  descriptions  of 

the  modules  (or  CSUs)  performing  that  function. 


